[People]
;Name=Function,Function details
Arnaud Bouchez=Project Manager,Develop software and manage associated projects

[Project]
Name=Synopse mORMot Framework
Company=Synopse
ReleaseVersion=
ReleaseDate=
Manager=Arnaud Bouchez
MainSection=DI
; so we first check for [DI] and [DILayout]
NoRiskDisplay=Not implemented
DestinationDir=D:\Documents\SynProject
; path to store all created .doc (not to be inside versioning tree)
OldWordOpen=No
; if OldWordOpen=Yes, Conversion is made visible on the screen (compatible with some Word 2000 installations)
DefLang=1033
Logo=logo.png
; this picture will be displayed top of every document front page
HeaderColWidth=22,37,22,19
; page header columns width, according to Manager=The Manager's name
NoConfidential=Yes
; so that no "Confidential" text will appear in page footer - seems convenient for a GPL document ;)
HeaderWithLogo=Yes
; custom page header with the synopse logo
HtmlSideBar=Overview/Meet the mORMot:SOURCE,Download/How to install:TITL_113,API Reference/Units and classes:SIDE_MORMOT_FRAMEWORK,FAQ/Frequently Asked Questions:TITL_123,Forum/Get support:https://synopse.info/forum,TimeLine/Open Source:https://synopse.info/fossil/timeline,Blog/Latest News:http://blog.synopse.info,Donate/Adopt a mORMot!:https://synopse.info/fossil/wiki?name=HelpDonate,Licence Terms/Either MPL, LGPL or GPL:TITL_34
; the sidebar first links, for html export

{\b Document License}
{\i Synopse mORMot Framework Documentation}.\line Copyright (C) 2008-2022 Arnaud Bouchez.\line Synopse Informatique - @https://synopse.info
The {\i Synopse mORMot Framework Source Code} is licensed under GPL / LGPL / MPL licensing terms, free to be included in any application.
;This documentation has been generated using {\i Synopse SynProject} - @https://synopse.info/fossil/wiki?name=SynProject
;This document is a free document; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 3 of the License, or (at your option) any later version.
The {\i Synopse mORMot Framework Documentation} is a free document, released under a GPL 3.0 License, distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
;You should have received a copy of the GNU General Public License along with this documentation. If not, see @http://www.gnu.org/licenses
{\b Trademark Notice}
Rather than indicating every occurrence of a trademarked name as such, this document uses the names only in an editorial fashion and to the benefit of the trademark owner with no intention of infringement of the trademark.

[Pictures]
SmartCalculator.png=278x328 35%,Smart Mobile Studio Calculator Sample
synfilevcl.png=1015x666 95%,User Interface generated using VCL components
synfiletms.png=955x610 95%,User Interface generated using TMS components
cartoon08.png=230x205 35%,Adopt a mORMot
cartoon07.png=250x213 35%,Adopt a mORMot
cartoon06.png=200x273 20%,Adopt a mORMot
cartoon05.png=250x234 30%,Adopt a mORMot
cartoon04.png=200x286 25%,Adopt a mORMot
cartoon03.png=250x285 30%,Adopt a mORMot
cartoon02.png=250x258 30%,Adopt a mORMot
cartoon01.png=371x240 30%,Adopt a mORMot
IamLost.png=300x283 30%,Meet the mORMot
logo.png=200x47 25%,Synopse Logo
; here are stored the Pictures properties, as PictureFileName=WIDTHxHEIGHT PERCENT%,Caption

[DI]
Owner=DI
Order=DI
Name=Design Input Product Specifications
ItemName=DI
DisplayName=Design Input
Purpose=Create high level description of software specifications
PreparedBy=Arnaud Bouchez
ReviewedBy=
ApprovedBy=
; Revision* multiple revision Table: ignored values are taken from current, older below
RevisionDescription=Initial Version
RevisionDate=
Revision=1.18
; [DILayout] list the global DI outline (lines beginning with : are titles)
; [DI-*] details all items
DefaultPreparedBy=Arnaud Bouchez
; all [DI-*] PreparedBy= default name
DocumentFrontPage=ProjectDetails,Warning,PeopleDetails,RevisionDetails,AddPurpose
;SubDocFrontPage=Warning,PeopleDetails

:System Specifications
; special lines begin with
;   :  for titles (no Name=Value pairs after a :title)
;   :# Title   for a later reference as @#@ (":1 Title" then @1@)
;   -  for a list item
;   !  for pascal source
;   !! for modified pascal source line
;   &  for c c++ source
;   &! for modified c c++ source line
;   #  for c# source
;   #! for modified c# source line
;   $  for text file (fixed-width font)
;   $! for modified text file line (fixed-width font)
;   %filename.jpg [640x480 85%] for images jpg/bmp/png/emf - see [Pictures]
;   %%FirmwareBoot for diagram images, i.e. not listed in [Pictures] but created with \graph FirmwareBoot ...
;   |%30%40%30   then |Col 1|Col 2|Col 3  for every row, ending with |%  for columns
;   |%=-30%40%30  -> =:no indent -:no border
;   =[SectionName] to inline the [SectionName] content at this place
; text can be formated as rtf (with \b \i { } e.g.) - each new text paragraph will be ended with \par
; {} can be used for a \par alone (void lines are just ignored)
; you can link to another item with @SectionName@ (@DI-4.1@ e.g) or @DocName@ (@SRS@) or @PeopleName@ (@A.Bouchez@) or either @%Picture.png@
; you can embedd a picture within a table cell e.g., by using @=%picture.png@ - in this case, this is not a "button"
; internet links will be handled as hyperlink, with @http://synopse.info
; in the [SDD-*] sections, specify @Module\filename.pas@ for each file name, @!Module\filename.pas@ for filename modified or @!procedurename!Module\filename.pas@ in order to specify the procedure name. The corresponding units (and procedures) will be highlited in the [SAD] document. Just click on the button to use the Object Browser window.
; some special lines commands can be entered:
;  \page        to force a new page
;  \landscape   to change the page orientation to landscape
;  \portrait    to change the page orientation to portrait
;  \footer blabla  to change the footer text
;  \Layout      to add a list with all DILayout titles
;  \LayoutPage  idem + associated pages in the document
;  \risk        to add the Risk Assessment Scale table
;  \Source      (for [SAD] section) to add the list of the Source=.. modules
;  \SourcePage  idem + associated pages in the document
;  \include filename.ext    ext will be used to append !&#$ left
;  \graph UniqueImageName [Title] then following lines either .dot normal text, or "\From Text\To Text[\Label between both]" - use F12 to have the dialog
;  \TableSoftwareChanges or \TableTraceabilityMatrix for SCRS
;  \TableNewFeatures or \TableBugFixes or \TableTests[=October 16, 2008] for Release Notes
;  \TableDI=6.3.1.2,6.3.1.3 for a table with all the supplied Design Inputs
;  \TableDocuments[=DI,SRS,SDD,SAD] for a table with the supplied document details
;  \Implements TableImplementsName #.# [Description][\DocumentName] (\Implements ISO 4.3 Software safety classification) in the text - points to the current document, or the specified DocumentName
;  \TableImplements=TableImplementsName (\TableImplements=ISO) to create the list, sorted by ascending #.# numbers, with description if any and corresponding document
;  =[SectionName] to include this section content at the current place
; in the [Test-*] sections, special auto-defined columns can be used with |Actions[|Expected Results] - manual tables can be used as usual (with |%..)
This document is intended to describe the Design Input Product Specifications.
: Definitions
{\b Added Value} - This level of achievement should be the target of the design team, because achieving this level of performance adds value to the product. However failure to achieve this level does not evoke additional management review.
{\b Must Have} - This level of achievement must be reached in the final design output. Because of possible negative financial impacts, if this level of performance is not achieved, management review will be triggered.
: Project Concept
:  Purpose and Scope
This document focuses on the {\i Synopse mORMot Framework} library.
The purpose of this @DI@ is to detail the marketing requirements/product specifications for the 1.18 release of the {\i Synopse mORMot Framework library}. The requirements and specifications found in this document are derived from customer market research, regulatory input and industry common practice.
:  Concept Statement
It was observed that a true JSON and RESTful oriented Client-Server framework was missing in the {\i Delphi} programing environment.
Latest versions of {\i Delphi} (i.e. {\i Delphi} 2010 and up) provide a JSON and RESTful mechanism named DataSnap (in the {\i Architect} or {\i Enterprise} editions), but such a feature could be implemented with previous versions of the {\i Delphi} compiler as well, with a more open architecture.
This framework shall use a innovative ORM (Object-relational mapping) approach, based on the RTTI (Runtime Type Information) provided by the {\i Delphi} language. It shall expose Server data access and business services to Clients, using JSON over several communication protocols.
After evaluation of most used database engines, the {\i SQLite3} engine was found out to be secure, fast, and perfectly adapted as a stand-alone database engine for this framework, able to access other remote database engines using its unique {\i Virtual Tables} mechanism.
Together with this Client-Server data and business architecture, a set of User Interface components (especially Database Grid and Reporting system) are provided within the framework.
The main approach of this framework is to avoid @*RAD@ in the development of projects. RAD has been proved to be a good candidate about prototyping, but is not the best approach for creating a robust and maintainable application. Best practices (as MVC, n-Tier or SOA) shall be used instead.
: Expected Use
Any application which need moderate database usage (up to some GB of data) with easy setup and administration, together with a secure @*ACID@ behavior in a Client-Server environment should consider using the {\i Synopse mORMot Framework}.
: Requirement Exceptions
This framework was developed in order to run mainly under any {\i Delphi} compiler, from version {\i Delphi} 6 to the latest Delphi version ({\i Delphi 11 Alexandria} at time of this writing).
On the {\i server side}, it targets both {\i Win32} and {\i Win64} platforms (using the 64-bit compiler included in latest {\i Delphi} XE2 and up).
For clients, in addition to those {\i Win32} / {\i Win64} platforms, you have cross-platform code generation abilities, for any {\i Delphi} or {\i @*FreePascal@} target (including {\i @*OSX@} and mobile {\i iOS} or {\i Android}), or AJAX / HTML5 clients via {\i @*Smart Mobile Studio@} - see @90@.
=[License]
\page
:Software Design Input
The Software @DI@ items follow these main divisions:
\LayoutPage

[DILayout]
; lines beginning with : will be titles for general DI layout - the 'DI-' chars are added before numbers listed below
:Client Server ORM/SOA framework
2.1.1
2.1.1.1
2.1.1.2
2.1.2
2.1.3
2.1.4
2.1.5
:SQlite3 engine
2.2.1
2.2.2
2.2.3
:User interface
2.3
2.3.1
2.3.2

[DI-2.1.1]
Risk=1,1,3,Arnaud Bouchez,Initial release
Request=Initial release
Ident=The framework shall be Client-Server oriented

[DI-2.1.1.1]
Risk=1,1,3,Arnaud Bouchez,Initial release
Request=Initial release
Ident=A RESTful mechanism shall be implemented

[DI-2.1.1.2]
Risk=1,1,3,Arnaud Bouchez,Initial release
Request=Initial release
Ident=Commmunication should be available directly in the same process memory, or remotly using Named Pipes, Windows messages or HTTP/1.1 protocols

[DI-2.1.2]
Risk=1,1,3,Arnaud Bouchez,Initial release
Request=Initial release
Ident=UTF-8 JSON format shall be used to communicate

[DI-2.1.3]
Risk=1,1,3,Arnaud Bouchez,Initial release
Request=Initial release
Ident=The framework shall use an innovative ORM (Object-relational mapping) approach, based on classes RTTI (Runtime Type Information)

[DI-2.1.4]
Risk=1,1,3,Arnaud Bouchez,Initial release
Request=Initial release
Ident=The framework shall provide some Cross-Cutting components

[DI-2.1.5]
Risk=1,1,3,Arnaud Bouchez,Initial release
Request=Initial release
Ident=The framework shall offer a complete SOA process

[DI-2.2.1]
Risk=1,1,3,Arnaud Bouchez,Initial release
Request=Initial release
Ident=The {\i SQLite3} engine shall be embedded to the framework

[DI-2.2.2]
Risk=1,1,3,Arnaud Bouchez,Initial release
Request=Initial release
Ident=The framework libraries, including all its {\i SQLite3} related features, shall be tested using Unitary testing

[DI-2.2.3]
Risk=1,1,3,Arnaud Bouchez,Initial release
Request=Initial release
Ident=The framework shall be able to access any external database, via OleDB, ODBC or direct access for Oracle (OCI) or SQLite3 (for external database files)

[DI-2.3]
Risk=1,1,3,Arnaud Bouchez,Initial release
Request=Initial release
Ident=User Interface and Report generation should be integrated

[DI-2.3.1]
Risk=1,1,3,Arnaud Bouchez,Initial release
Request=Initial release
Ident=An User Interface, with buttons and toolbars shall be easily being created from the code, with no RAD needed, using RTTI and data auto-description

[DI-2.3.2]
Risk=1,1,3,Arnaud Bouchez,Initial release
Request=Initial release
Ident=A reporting feature, with full preview and export as PDF or TXT files, shall be integrated

[RK]
; self-owned Risk Asssessment document
Owner=RK
Order=RK
ItemName=FMEA
DisplayName=Design FMEA
Name=Design FMEA File
DocName=Design FMEA File
; this DocName will be used for generating .DOC filename (instead of ItemName)
Purpose=List {\i Failure Modes and Effects Analysis} (FMEA)
PreparedBy=Arnaud Bouchez
ReviewedBy=
ApprovedBy=
Revision=1.18
RevisionDate=
RevisionDescription=Initial Version
; Revision* multiple revision Table: ignored values are taken from current, older below
DocumentFrontPage=ProjectDetails,Warning,PeopleDetails,RevisionDetails,AddPurpose
WriteRisk=Yes
; Write Risk assessment table summary after every RK item
WriteTableOfContent=Yes
; Write global Table Of Contents
TableOfContentsAtTheBeginning=Yes
; if the Table of Contents must be at the beginning (default=No=at the end of the file)
DocumentIndex=Pictures,Implements ISO=ISO 123456 requirements
; "Pictures" will create a table with all picture appearing in this document
; "Implements ISO=..." will create a table with all appearing "\Implements ISO 3.4" pages, with the specified item name

:Introduction
The @RK@ is a reference document used to list the {\i Failure Modes and Effects Analysis} (FMEA) identified for the {\i Synopse mORMot Framework} library.
The "{\i Failure modes and effects analysis}" (FMEA) is a procedure in operations management for analysis of potential failure modes within a system for classification by severity or determination of the effect of failures on the system. {\i Failure modes} are any errors or defects in a process, design, or item, especially those that affect the customer, and can be potential or actual. {\i Effects analysis} refers to studying the consequences of those failures.
In practice, a Risk Assessment team starts with a block diagram of a system. The team then considers what happens if each block of the diagram fails, and fills in a table in which failures are paired with their effects and an evaluation of the effects. The design of the system is then corrected, and the table adjusted until the system is known not to have unacceptable problems.
This @RK@ lists most FMEA items identified as possible Software Failure for the {\i Synopse mORMot Framework}.
: Risk Assessment
In the following @RK@, a numerical Risk Assessment is given for every FMEA item, according to the {\i Risk Assessment Scale} table below.
A summary explanation is indicated, together with the names of those who made each evaluation.
\risk
: Responsibilities
- Synopse will try to correct any identified issue;
- The Open Source community will create tickets in a public Tracker web site located at @https://synopse.info/fossil ;
- Synopse work on the framework is distributed without any warranty, according to the chosen license terms;
- This documentation is released under the GPL (GNU General Public License) terms, without any warranty of any kind.
\page
:FMEA
: Fault Tree
Here is the Fault Tree of the framework, displayed in a graphical way:
\graph FTA mORMot Framework Fault Tree
\mORMot Framework\Framework Architecture
\Framework Architecture\Invalid Concurent Access
\Invalid Concurent Access\Database corruption
\Invalid Concurent Access\Wrong Client-Server synchro
\Wrong Client-Server synchro\Enduser problems
\Framework Architecture\Main Server Crashed
\Framework Architecture\Security issue
\Security issue\Enduser problems
\Security issue\Database corruption
\Main Server Crashed\Database corruption
\Database corruption\Enduser problems
\mORMot Framework\User Interface
\User Interface\Security issue
\User Interface\Inconsistent Layout
\Inconsistent Layout\Timeout problems
\Inconsistent Layout\Incorrect User action
\Incorrect User action\Enduser problems
\User Interface\Function not working
\Function not working\Timeout problems
\

[SRS]
Owner=DI
Order=SRS
; Owner: [SRS-*] -> * reference; Order=SRS -> [SRS-*] have sub items
;Refers=RK
; Refers will add all [SRS-RK-*] items to the list, after the DI
Name=Software Requirements Specifications
ItemName=SWRS
Purpose=Interpret design inputs and specify software design features
PreparedBy=Arnaud Bouchez
ReviewedBy=
ApprovedBy=
Revision=1.18
RevisionDate=
RevisionDescription=Initial Version
; Revision* multiple revision Table: ignored values are taken from current, older below
;PreparedBy=..   ignored values are taken from current
; [SRS-*] sections describe each item ([DI] items + other items)
; [SRS-*] are displayed as they appear in the file
DocumentFrontPage=ProjectDetails,Warning,PeopleDetails,RevisionDetails,AddPurpose
WriteRisk=Yes
; Write Risk assessment table summary after every DI
WriteTableOfContent=Yes
; Write global Table Of Contents at the end of the file
TableOfContentsAtTheBeginning=Yes
; if the Table of Contents must be at the beginning (default=No=at the end of the file)
DocumentIndex=Pictures,Implements ISO=ISO 123456 requirements

:Introduction
: Documentation overview
The whole Software documentation process follows the typical steps of this diagram:
\graph FMEADI Design Inputs, FMEA and Risk Specifications
\User¤Requirements\Design Inputs¤(DI)\define
\Regulatory¤Requirements\Design Inputs¤(DI)
\Design Inputs¤(DI)\Specifications¤(SWRS)\are specified by
\System-wide¤Risk Assessment\SW FMEA¤(RK)\defines
\SW FMEA¤(RK)\Specifications¤(SWRS)
\Specifications¤(SWRS)\Architecture + Design¤(SAD+SDD)\is implemented by
\Architecture + Design¤(SAD+SDD)\Test + Documentation\is associated to
\Test + Documentation\Specifications¤(SWRS)\refers to
\
: Purpose
This @SRS@ applies to the first public release of the {\i Synopse mORMot Framework}.
It describes the software implementation of each design input as specified by the @DI@.
The sections of this document follow the @DI@ divisions:
\LayoutPage
;Then all items created from the @RK@ are listed:
;\referspage
For each Design Input item, the corresponding justification is specified, between parenthesis (SCR #65, e.g.).
Every @SRS@ item is named about the corresponding @DI@ item, or, in case the initial {\i Design Input} is too large and must be divided into some {\i SWRS} more precise items, an unique name is proposed.
: Risk Assessment
The Risk assessment indicated below was evaluated as a team work, based on the software solution proposed.
In the following @SRS@, a numerical Risk Assessment is given for every Design Input item, according to the {\i Risk Assessment Scale} table below.
A summary explanation is indicated, together with the names of those who made each evaluation.
\risk
: Responsibilities
- Synopse will try to correct any identified issue;
- The Open Source community will create tickets in a public Tracker web site located at @https://synopse.info/fossil ;
- Synopse work on the framework is distributed without any warranty, according to the chosen license terms;
- This documentation is released under the GPL (GNU General Public License) terms, without any warranty of any kind.

[SRS-DI-2.1.1]
; DI-2.1.1 - The framework shall be Client-Server oriented
ShortName=Client-Server framework

Client–Server model of computing is a distributed application structure that partitions tasks or workloads between service providers, called servers, and service requesters, called clients.
Often clients and servers communicate over a computer network on separate hardware, but both client and server may reside in the same system. A server machine is a host that is running one or more server programs which share its resources with clients. A client does not share any of its resources, but requests a server's content or service function. Clients therefore initiate communication sessions with servers which await (listen for) incoming requests.
The {\i Synopse mORMot Framework} shall implement such a Client-Server model by a set of dedicated classes, over various communication protocols, but in an unified way. Application shall easily change the protocol used, just by adjusting the class type used in the client code. By design, the only requirement is that protocols and associated parameters are expected to match between the Client and the Server.

[SRS-DI-2.1.1.1]
; DI-2.1.1.1 - A RESTful mechanism shall be implemented
ShortName=RESTful framework

REST-style architectures consist of clients and servers, as was stated in @SRS-DI-2.1.1@. Clients initiate requests to servers; servers process requests and return appropriate responses. Requests and responses are built around the transfer of "representations" of "resources". A resource can be essentially any coherent and meaningful concept that may be addressed. A representation of a resource is typically a document that captures the current or intended state of a resource.
In the {\i Synopse mORMot Framework}, so called "resources" are individual records of the underlying database, or list of individual fields values extracted from these databases, by a SQL-like query statement.

[SRS-DI-2.1.1.2]
; DI-2.1.1.2 - Commmunication should be available directly in the same process memory, or remotly using Named Pipes, Windows messages or HTTP/1.1 protocols
ShortName=Communication via diverse protocols

In computing and telecommunications, a protocol or communications protocol is a formal description of message formats and the rules for exchanging those messages.
The {\i Synopse mORMot Framework} shall support the following protocols for remote access, according to the Client-Server architecture defined in @SRS-DI-2.1.1@:
- Direct in-process communication;
- Using Windows Messages;
- Using Named pipe;
- Using HTTP/1.1 over TCP/IP.

[SRS-DI-2.1.1.2.1]
Parent=DI-2.1.1.2
Ident=Client-Server Direct communication shall be available inside the same process
ShortName=In-Process communication

[SRS-DI-2.1.1.2.2]
Parent=DI-2.1.1.2
Ident=Client-Server Named Pipe communication shall be made available by some dedicated classes
ShortName=Named Pipe protocol

[SRS-DI-2.1.1.2.3]
Parent=DI-2.1.1.2
Ident=Client-Server Windows Messages communication shall be made available by some dedicated classes
ShortName=Windows Messages protocol

[SRS-DI-2.1.1.2.4]
Parent=DI-2.1.1.2
Ident=Client-Server HTTP/1.1 over TCP/IP protocol communication shall be made available by some dedicated classes, and ready to be accessed from outside any {\i Delphi} Client (e.g. the implement should be AJAX ready)
ShortName=HTTP/1.1 protocol

[SRS-DI-2.1.2]
; DI-2.1.2 - UTF-8 JSON format shall be used to communicate

JSON, as defined in the @SAD@, is used in the {\i Synopse mORMot Framework} for all Client-Server communication. JSON (an acronym for {\i JavaScript} Object Notation) is a lightweight text-based open standard designed for human-readable data interchange. Despite its relationship to {\i JavaScript}, it is language-independent, with parsers available for virtually every programming language.
JSON shall be used in the framework for returning individual database record content, in a disposition which could make it compatible with direct {\i JavaScript} interpretation (i.e. easily creating {\i JavaScript} object from JSON content, in order to facilitate AJAX application development). From the Client to the Server, record content is also JSON-encoded, in order to be easily interpreted by the Server, which will convert the supplied field values into proper SQL content, ready to be inserted to the underlying database.
JSON should be used also within the transmission of request rows of data. It therefore provide an easy way of data formating between the Client and the Server.
The {\i Synopse mORMot Framework} shall use UTF-8 encoding for the character transmission inside its JSON content. UTF-8 (8-bit Unicode Transformation Format) is a variable-length character encoding for Unicode. UTF-8 encodes each character (code point) in 1 to 4 octets (8-bit bytes). The first 128 characters of the Unicode character set (which correspond directly to the ASCII) use a single octet with the same binary value as in ASCII. Therefore, UTF-8 can encode any Unicode character, avoiding the need to figure out and set a "code page" or otherwise indicate what character set is in use, and allowing output in multiple languages at the same time. For many languages there has been more than one single-byte encoding in usage, so even knowing the language was insufficient information to display it correctly.

[SRS-DI-2.1.3]
; DI-2.1.3 - The framework shall use an innovative ORM (Object-relational mapping) approach, based on classes RTTI (Runtime Type Information)

ORM, as defined in the @SAD@, is used in the {\i Synopse mORMot Framework} for accessing data record fields directly from {\i Delphi} Code.
Object-relational mapping (ORM, O/RM, and O/R mapping) is a programming technique for converting data between incompatible type systems in relational databases and object-oriented programming languages. This creates, in effect, a "virtual object database" that can be used from within the {\i Delphi} programming language.
The {\f1\fs20 published} properties of classes inheriting from a new generic type named {\f1\fs20 TSQLRecord} are used to define the field properties of the data. Accessing database records (for reading or update) shall be made by using these classes properties, and some dedicated Client-side methods.

[SRS-DI-2.1.4]
; DI-2.1.4 - The framework shall provide some Cross-Cutting components

{\i Cross-Cutting infrastructure layers} shall be made available for handling data filtering and validation, security, session, cache, logging and testing (framework uses test-driven approach and features stubbing and mocking).
All crosscutting scenarios are coupled, so you benefit of consisting APIs and documentation, a lot of code-reuse, JSON/RESTful orientation from the ground up.

[SRS-DI-2.1.5]
; DI-2.1.5 - The framework shall offer a complete SOA process

In order to follow a {\i Service Oriented Architecture} design, your application's business logic can be implemented in several ways using {\i mORMot}:
- Via some {\f1\fs20 @*TSQLRecord@} inherited classes, inserted into the database {\i model}, and accessible via some @*REST@ful URI - this is implemented by our @*ORM@ architecture - see @SRS-DI-2.1.3@;
- By some RESTful @**service@s, implemented in the Server as {\i published methods}, and consumed in the Client via native {\i Delphi} methods;
- Defining some RESTful {\i service @*contract@s} as standard {\i Delphi} {\f1\fs20 interface}, and then run it seamlesly on both client and client sides.

[SRS-DI-2.2.1]
; DI-2.2.1 - The SQLite3 engine shall be embedded to the framework

The {\i SQLite3} database engine is used in the {\i Synopse mORMot Framework} as its kernel database engine. {\i SQLite3} is an ACID-compliant embedded relational database management system contained in a C programming library.
This library shall be linked statically to the {\i Synopse mORMot Framework}, or using official external {\f1\fs20 sqlite3.dll} distribution, and interact directly from the {\i Delphi} application process.
The {\i Synopse mORMot Framework} shall enhance the standard {\i SQLite3} database engine by introducing some new features stated in the @SAD@, related to the Client-Server purpose or the framework - see @SRS-DI-2.1.1@.

[SRS-DI-2.2.2]
; DI-2.2.2 - The framework libraries, including all its {\i SQLite3} related features, shall be tested using Unitary testing

The {\i Synopse mORMot Framework} shall use all integrated Unitary testing features provided by a common testing framework integrated to all Synopse products. This testing shall be defined by classes, in which individual published methods define the actual testing of most framework features.
All testing shall be run at once, for example before any software release, or after any modification to the framework code, in order to avoid most regression bug.

[SRS-DI-2.2.3]
; DI-2.2.3 - The framework shall be able to access any external database, via OleDB or direct access for Oracle (OCI) or SQLite3 (for external database files)

The following external database providers shall be made available to the framework ORM:
- Any {\i OleDB} provider;
- Any {\i ODBC} provider;
- Any {\f1\fs20 TDataset} / {\f1\fs20 DB.pas} provider;
- {\i Oracle} database, via direct OCI client access;
- {\i SQlite3} database engine.
A dedicated set of classes shall implement access, together with some advanced syntaxic sugar, like fast late-binding, or advanced ORM mechanism, like Virtual Tables.

[SRS-DI-2.3]
; DI-2.3 - User Interface and Report generation should be integrated

The {\i Synopse mORMot Framework} shall provide User Interface and Report generation from code.
Such a ribbon-oriented interface shall be made available, in a per-table approach, and associated reports.
Here is a sample of screen content, using proprietary TMS components:
%synfiletms.png
And here is the same application compiled using only VCL components, available from {\i Delphi} 6 up to the latest available {\i Delphi} version:
%synfilevcl.png

[SRS-DI-2.3.1]
; DI-2.3.1 - An User Interface, with buttons and toolbars shall be easily being created from the code, with no RAD needed, using RTTI and data auto-description

The {\i Synopse mORMot Framework} shall provide some User-Interface dedicated units, allowing database grid display, on screen tool-bar creation (by an internal system of actions using {\i Delphi} RTTI - see the corresponding paragraph in the @SAD@), and integrated reporting - see @SRS-DI-2.3.2@.
No RAD approach is to be provided: the Client application User Interface will be designed not by putting components on the IDE screen, but directly from code.

[SRS-DI-2.3.1.1]
Parent=DI-2.3.1
Ident=A Database Grid shall be made available to provide data browsing in the Client Application - it shall handle easy browsing, by column resizing and sorting, on the fly customization of the cell content
ShortName=Database Grid Display

[SRS-DI-2.3.1.2]
Parent=DI-2.3.1
Ident=Toolbars shall be able to be created from code, using RTTI and enumerations types for defining the action
ShortName=RTTI generated Toolbars

[SRS-DI-2.3.1.3]
Parent=DI-2.3.1
Ident=Internationalization (i18n) of the whole User Interface shall be made available by defined some external text files: {\i Delphi} resourcestring shall be translatable on the fly, custom window dialogs automaticaly translated before their display, and User Interface generated from RTTI should be included in this i18n mechanism
ShortName=Automated i18n

[SRS-DI-2.3.2]
; DI-2.3.2 - A reporting feature, with full preview and export as PDF or TXT files, shall be integrated

The {\i Synopse mORMot Framework} shall provide a reporting feature, which could be used stand-alone, or linked to its database mechanism. Reports shall not be created using a RAD approach (e.g. defining bands and fields with the mouse on the IDE), but shall be defined from code, by using some dedicated methods, adding text, tables or pictures to the report. Therefore, any kind of report shall be generated.
This reports shall be previewed on screen, and exported as PDF or TXT on request.

[Risk]
Owner=SRS
Order=SRS
; Owner: [Risk-*] -> * reference; Order=SRS -> [Test-*] have no sub items
Name=Software Implementation Risk Assessment
DocName=Risk Assessment
; this DocName will be used for generating .DOC filename (instead of ItemName)
Purpose=Perform a risk assessment of the SWRS implementation
PreparedBy=Arnaud Bouchez
ReviewedBy=
ApprovedBy=
Revision=1.18
RevisionDate=
RevisionDescription=Initial Version
; Revision* multiple revision Table: ignored values are taken from current, older below
; rev.1 summary values are in [DI] Risk=Severity,Probability,Occurence,Comment
; [Risk-*] sections contain rev.2 details for each SRS, [Risk-SER-03] or [Risk-4.5.1] e.g.
DocumentFrontPage=ProjectDetails,Warning,PeopleDetails,RevisionDetails,AddPurpose,RiskTable

[SAD]
Owner=SRS
Order=SRS
; Owner: [SAD-*] -> * reference; Order=SRS -> [SAD-*] have no sub items
Name=Software Architecture Design
Purpose=Describe the implications of each software requirement specification on all the affected software modules
PreparedBy=Arnaud Bouchez
ReviewedBy=
ApprovedBy=
Revision=1.18
RevisionDate=
RevisionDescription=Initial Version
; Revision* multiple revision Table: ignored values are taken from current, older below
; [SAD-*] sections contain details for each SRS, [SAD-SER-03] or [SAD-DI-4.10.6] e.g.
; [SAD-*] are displayed as they appear in the [SRS-*] sections
DocumentFrontPage=ProjectDetails,Warning,PeopleDetails,RevisionDetails,AddPurpose
Source=Main,SynFile
; presence of Source=.. adds global description for each SourceFile=.. and for @Module\filename.pas@ for the project (TProject.Parse: array of TSection - [SAD-Module] will get files from @Module\name.pas@, e.g.)
SourceSDD=SDD
; name of the [SDD] section to be parsed for modified files as @!Module\bidule.pas@
DefaultPath=D:\Dev
; global default directory to trim in the documentation, and used by default in [SAD-module] SourcePath=..
Directives=INCLUDE_FTS3
; optional global conditional define when parsing the source
WriteTableOfContent=Yes
; Write global Table Of Contents of the file
TableOfContentsAtTheBeginning=Yes
; if the Table of Contents must be at the beginning (default=No=at the end of the file)
DocumentIndex=Pictures,Source,Index
; "Source" will create a table with all @source.code@ files within this document
; "Index" an index of all @*keyword@, Picture of all %pictures
WithAllfields=Yes
; write all field properties of all object and classes: document is huge, but contains all available architecture intel

:Foreword
The whole Software documentation process follows the typical steps of this diagram:
%%FMEADI
: Purpose
This @SAD@ applies to the 1.18 release of the {\i Synopse mORMot Framework} library.
After a deep presentation of the framework architecture and main features, each source code unit is detailed, with clear diagrams and tables showing the dependencies between the units, and the class hierarchy of the objects implemented within.
The {\i SynFile} main demo is presented on its own, and can be used as a general {\i User Guide} of its basic ORM features and User Interface generation - see @50@.
At the end of this document, @SRS@ items are linked directly to the class or function involved with the @SDD@, from the source code.
: Responsibilities
- Support is available in the project forum - @https://synopse.info/forum - from the {\i mORMot} Open Source community;
- Tickets can be created in a public Tracker web site located at @https://synopse.info/fossil , which publishes also the latest version of the project source code;
- Synopse can provide additional support, expertise or enhancements, on request;
- Synopse work on the framework is distributed without any warranty, according to the chosen license terms - see @34@;
- This documentation is released under the GPL ({\i GNU @*General Public License@}) terms, without any warranty of any kind.
=[GPL]

[SAD-Source]
; this Section body is added as the introduction of the document first part
TitleOffset=0
DisplayName=mORMot Framework Overview

:Synopse mORMot Overview
%IamLost.png
{\i Synopse mORMot} is an Open Source @*Client-Server@ @*ORM@ @*SOA@ @*MVC@ framework for {\i Delphi} 6 up to the latest available {\i Delphi} version and @*FPC@ 3.2, targeting {\i Win/@*Linux@} for the server, and any platform for clients (including mobile or AJAX).
The main features of {\i mORMot} are therefore:
- {\i ORM/ODM}: objects persistence on almost any database (SQL or NoSQL);
- {\i SOA}: organize your business logic into @*REST@ services;
- {\i Clients}: consume your data or services from any platform, via ORM classes or SOA interfaces;
- {\i Web MVC}: publish your ORM/SOA process as responsive @*Web Application@s.
With local or remote access, via an auto-configuring Client-Server @*REST@ design.
\graph mORMotDesignORMSOA General mORMot architecture
subgraph cluster_0 {
"SQLDB";
label="SQL Databases";
}
subgraph cluster_1 {
"NoSQLDB";
label="NoSQL Databases";
}
subgraph cluster_2 {
"Services";
label="Services";
}
subgraph cluster_3 {
\SQLDB\ORM
\NoSQLDB\ODM
\REST Server\MVC/MVVM¤Web Server
\ORM\REST Server
\ODM\REST Server
\Services\SOA
\SOA\REST Server
label=" mORMot\nServer";
}
\REST Server\Stand Alone¤Application
subgraph cluster_4 {
label="REST   Clients";
\REST Server\... any
\REST Server\AJAX
\REST Server\Mobile
\REST Server\Delphi
}
subgraph cluster_5 {
label="    Web   Clients";
\MVC/MVVM¤Web Server\Desktop
\MVC/MVVM¤Web Server\ Mobile
}
subgraph cluster_6 {
label="         Featuring";
"Featured";
}
=SQLDB=SQLite3 - Firebird - NexusDB\nPostgreSQL - MySQL - DB2\nMS SQL - Oracle - Informix
=NoSQLDB=MongoDB\nIn-Memory\nFiles
=Services=Method-based Services\nInterface-based Services\nAsynchronous (Push) Services\nRemote (Saas) Services
=Featured=User Management - Security & Rights - Sessions - Replication¤Unit Testing - Mocks/Stubs - Logging - Performance - Profiling¤http.sys - WebSockets - MultiCore - Templates (MVC) ¤JSON - JavaScript Engine -  Reporting - PDF - UI
\
{\i mORMot} offers all features needed for building any kind of modern software project, with state-of-the-art integrated software components, designed for both completeness and complementarity, offering {\i @*convention over configuration@} solutions, and implemented for speed and efficiency.
For {\i storing some data}, you define a {\f1\fs20 class}, and the framework will take care of everything: routing, JSON marshalling, table creation, SQL generation, validation.
For {\i creating a service}, you define an {\f1\fs20 interface} and a {\f1\fs20 class}, and you are done. Of course, the same ORM/ODM or SOA methods will run on both server and client sides: code once, use everywhere!
For {\i building a MVC web site}, write a Controller class in Delphi, then some HTML Views using {\i @*Mustache@} templates, leveraging the same ORM/ODM or SOA methods as Model.
If you need a HTTP server, a proxy @*redirection@, master/slave @*replication@, @*publish-subscribe@, a test, a mock, add security, define users or manage rights, a script engine, a report, User Interface, switch to XML format or publish HTML dynamic pages - just pick up the right {\f1\fs20 class} or method. If you need a tool or feature, it is probably already there, waiting for you to use it.
The table content of this document makes it clear: this is no ordinary piece of software.
The {\i mORMot} framework provides an Open Source {\i self-sufficient set of units} (even {\i Delphi} starter edition is enough) for creating any {\i Multi-@*tier@} application, up to the most complex {\i @*Domain-Driven@} design - see @54@:
- {\i Presentation layer} featuring @*MVC@ UI generation with @*i18n@ and reporting for rich {\i Delphi} clients, {\i @*Mustache@}-based templates for web views - see @108@ - or rich @*AJAX@ clients;
- {\i Application layer} implementing Service Oriented Architecture via {\f1\fs20 interface}-based services (like @*WCF@) and Client-Server ORM - following a @*REST@ful model using @*JSON@ over several communication protocols (e.g. @*HTTP@/1.1 and @*HTTPS@);
- {\i Domain Model layer} handling all the needed business logic in plain {\i Delphi} objects, including high-level managed types like @*dynamic array@s or records for {\i Value Objects}, dedicated classes for {\i Entities} or {\i Aggregates}, and {\f1\fs20 variant} storage with late-binding for dynamic documents - your business logic may also be completed in {\i JavaScript} on the server side as stated @79@;
- {\i Data persistence infrastructure layer} with ORM persistence on direct @*Oracle@, @*MS SQL@, @*OleDB@, @*ODBC@, @*Zeos@ connection or any {\f1\fs20 DB.pas} provider (e.g. @*NexusDB@, @*DBExpress@, @*FireDAC@, @*AnyDAC@, @*UniDAC@...), with a powerful @*SQLite3@ kernel, and direct @*SQL@ access if needed - including SQL auto-generation for {\i SQLite3, Oracle, @*Jet/MSAccess@, MS SQL, @*Firebird@, @*DB2@, @*PostgreSQL@, @*MySQL@, @*Informix@} and {\i NexusDB} - the ORM is also able to use @*NoSQL@ engines via a native {\i @*MongoDB@} connection, for ODM persistence;
- {\i Cross-Cutting infrastructure layers} for handling data filtering and validation, @*security@, @*session@, @*cache@, logging and @*test@ing (framework uses @*test-driven@ approach and features @*stub@bing and @*mock@ing).
If you do not know some of those concepts, don't worry: this document will detail them - see @40@.
With {\i mORMot}, {\i ORM} is not used only for data persistence of objects in databases (like in other implementations), but as part of a global n-@*Tier@, Service Oriented Architecture (SOA), ready to implement {\i Domain-Driven} solutions.\line {\i mORMot} is not another ORM on which a transmission layer has been added, like almost everything existing in Delphi, C# or Java: this is a full Client-Server ORM/SOA from the ground up. This really makes the difference.
The business logic of your applications will be easily exposed as {\i Services}, and will be accessible from light clients (written in {\i Delphi} or any other mean, including AJAX).
The framework Core is non-visual: it provides only a set of classes to be used from code. But you have also some UI units available (including screen auto-creation, reporting and ribbon GUI), and you can use it from any RAD, web, or AJAX clients.
No dependency is needed at the client side (no DB driver, or third-party runtime): it is able to connect via standard HTTP or HTTPS, even through a corporate proxy or a VPN. Rich {\i Delphi} clients can be deployed just by copying and running a @*stand-alone@ small executable, with no installation process. Client authentication is performed via several secure methods, and communication can be encrypted via HTTS or with a proprietary SHA/@*AES@-256 algorithm. SOA endpoints are configured automatically for each published interface on both server and client sides, and creating a load-balancing proxy is a matter of one method call. Changing from one database engine to another is just a matter of one line of code; full audit-trail history is available, if needed, to track all changes of any class persisted by the ORM/ODM.
Cross-platform clients can be easily created, as {\i Win32} and {\i Win64} executables of course, but also for any platform supported by the {\i Delphi} compiler (including {\i Mac @*OSX@}, {\i @*iPhone@}/{\i @*iPad@} and {\i @*Android@}), or by {\i @*FreePascal@} / {\i @*Lazarus@}. AJAX applications can easily be created via {\i @*Smart Mobile Studio@}, as will any mobile operating system be accessible as an HTML5 web rich client or stand-alone {\i @*PhoneGap@} application, ready to be added to the {\i Windows}, {\i Apple} or {\i Google} store. See @86@ for how {\i mORMot} client code generation leverages all platforms.
Speed and scalability has been implemented from the ground up - see @59@: a genuine optimized multi-threaded core let a single server handle more than 50,000 concurrent clients, faster than {\i DataSnap}, {\f1\fs20 WCF} or {\f1\fs20 node.js}, and our rich SOA design is able to implement both vertical and horizontal scalable @*hosting@, using recognized enterprise-level SQL or NoSQL databases for storage.
In short, with {\i mORMot}, your ROI is maximized.
\page
:41 Client-Server ORM/SOA framework
The {\i Synopse mORMot framework} implements a Client-Server @*REST@ful architecture, trying to follow some MVC, N-@*Tier@, ORM, SOA best-practice patterns - see @40@.
Several clients, can access to the same remote or local server, using diverse communication protocols:
\graph mORMotDesign1 General mORMot architecture - Client / Server
\Internet (VPN)\Local Network
node [shape=box];
\Local Network\Server
subgraph cluster_0 {
"Client 1\n(Delphi)";
label="PC 1";
}
subgraph cluster_1 {
"Client 2\n(AJAX)";
label="PC 2";
}
subgraph cluster_2 {
\Client 4¤(Delphi)\Server
\Server\Client 4¤(Delphi)\JSON + REST¤named pipe¤connection
label="PC Server";
}
subgraph cluster_3 {
"Client n\n(Delphi)";
label="PC n";
}
subgraph cluster_4 {
"Client 3\n(Delphi)";
label="PC 3";
}
\Client 1¤(Delphi)\Local Network\JSON + REST¤over HTTP/1.1
\Client 2¤(AJAX)\Internet (VPN)\JSON + REST¤over HTTP/1.1
\Client 3¤(Delphi)\Local Network
\Client n¤(Delphi)\Internet (VPN)
\
Or the application can be @*stand-alone@:
\graph mORMotDesign2 General mORMot architecture - Stand-alone application
rankdir=LR;
node [shape=box];
subgraph cluster {
\Client\Server\direct ¤access
\Server\Client
label="Stand-Alone application";
}
\
Switch from this embedded architecture to the Client-Server one is just a matter of how {\i mORMot} classes are initialized. For instance, the very same executable can even be running as a stand-alone application, a server, or a client, depending on some run-time parameters!
\page
: Highlights
At first, some points can be highlighted, which make this framework distinct to other available solutions:
- @*Client-Server@ orientation, with optimized request caching and intelligent update over a @*REST@ful architecture - but can be used in stand-alone applications;
- No @*RAD@ components, but true @*ORM@ and @*SOA@ approach;
- Multi-@*Tier@ architecture, with integrated @*Business rules@ as fast ORM-based classes and {\i @*Domain-Driven@} design;
- {\i Service-Oriented-Architecture} model, using custom RESTful JSON services - you can send as JSON any {\f1\fs20 TStrings, TCollection, TPersistent} or {\f1\fs20 TObject} (via registration of a custom serializer) instance, or even a {\i dynamic array}, or any record content, with integrated JSON @*serialization@, via an @*interface@-based contract shared on both client and server sides;
- Truly RESTful @*authentication@ with a dual @*security@ model (session + per-query);
- Very fast @*JSON@ producer and parser, with caching at SQL level;
- Fast a configuration-less @*HTTP@  / @*HTTPS@ server using {\i @*http.sys@} kernel-mode server - but may communicate via named pipes, Windows Messages or in-process as lighter alternatives;
- Using {\i @*SQLite3@} as its kernel, but able to connect to any other database (via @*OleDB@ / @*ODBC@ / @*Zeos@ or direct client library access e.g. for @*Oracle@) - the {\f1\fs20 SynDB.pas} classes are self-sufficient, and do not depend on the {\i Delphi} {\f1\fs20 DB.pas} unit nor any third-party (so even the {\i Delphi} Starter edition is enough) - but the {\f1\fs20 SynDBDataset} unit is also available to access any {\f1\fs20 DB.pas} based solution (e.g. @*NexusDB@, @*DBExpress@, @*FireDAC@, @*AnyDAC@, @*UniDAC@ or even the @*BDE@...);
- RESTful ORM access to a @*NoSQL@ database engine like {\i @*MongoDB@} with the same code base;
- Ability to use @*SQL@ and RESTful requests over multiple databases at once (thanks to {\i SQLite3} unique @*Virtual Table@s mechanism);
- Full @*Text Search@ engine included, with enhanced Google-like ranking algorithm;
- Server-side @*JavaScript@ engine, for defining your business intelligence;
- Direct User Interface generation: grids are created on the fly, together with a modern Ribbon ('Office 2007'-like) screen layout - the code just has to define actions, and assign them to the tables, in order to construct the whole interface from a few lines of code, without any IDE usage;
- Integrated @*Report@ing system, which could serve complex @*PDF@ reports from your application;
- Designed to be as fast as possible (asm used when needed, buffered reading and writing avoid most memory consumption, multi-thread ready architecture...) so benchmarks sound impressive when compared to other solutions - see @59@;
- More than 1800 pages of documentation;
- {\i Delphi}, {\i FreePascal}, mobile and @*AJAX@ clients can share the same server, and ORM/SOA client access code can be generated on request for any kind of application - see @86@;
- Full source code provided - so you can enhance it to fulfill any need;
- Works from {\i Delphi} 6 up to the latest available {\i Delphi} version and FPC 3.2.x, truly Unicode (uses @*UTF-8@ encoding in its kernel, just like JSON), with any version of {\i Delphi} (no need to upgrade your IDE).
\page
: Benefits
As you can see from the previous section, {\i mORMot} provides a comprehensive set of features that can help you to manage your crosscutting concerns though a reusable set of components and core functionality.
%IamLost.png
Of course, like many developers, you may suffer from the well-known NIH ("Not Invented Here") syndrome. On the other side, it is a commonly accepted fact that the use of standard and proven code libraries and components can save development time, minimize costs, reduce the use of precious test resources, and decrease the overall maintenance effort.
Benefits of {\i mORMot} are therefore:
- @*KISS@ {\i convention over configuration} design: you have all needed features at hand, but with only one way of doing it - less configuration and less confusion for the developer and its customers;
- Pascal oriented: implementation is not following existing Java or C# patterns (with generics (ab)use, variable syntaxes and black-box approach), but try to unleash the object pascal genius;
- Integrated: all crosscutting scenarios are coupled, so you benefit of consisting APIs and documentation, a lot of code-reuse, JSON/RESTful orientation from the ground up;
- Tested: most of the framework is @*test-driven@, and all regression tests are provided, including system-wide integration tests;
- Do-not-reinvent-the-wheel, since we did it for you: it is now time to focus on your business;
- Open Source, documented and maintained: project is developed since years, with some active members - {\i mORMot} won't leave you soon!
\page
:66 Legacy code and existing projects
Even if {\i mORMot} will be more easily used in a project designed from scratch, it fits very well the purpose of evolving any existing {\i Delphi} project, or creating the server side part of an AJAX application.\line One benefit of such a framework is to facilitate the transition from a traditional @*Client-Server@ architecture to a N-@*Tier@ layered pattern.
Due to its modular design, you can integrate some framework bricks to your existing application:
- You may add logging to your code - see @16@, to track unresolved issues, and add customer-side performance profiling;
- Use low-level classes like {\f1\fs20 record} or {\i dynamic array} wrappers - see @48@, or our dynamic document storage via {\f1\fs20 variant} - see @80@, including @*JSON@ or binary persistence;
- You can use the direct DB layers, including the {\f1\fs20 @*TQuery@} emulation class - see @133@ - to replace some @**BDE@ queries, or introduce nice unique features like direct database access or {\i @*array bind@ing} for very fast data insertion - see @27@, or switch to a @*NoSQL@ database - see @83@;
- Reports could benefit of the {\f1\fs20 mORMotReport.pas} code-based system, which is very easy to use even on the server side (serving @*PDF@ files), when your business logic heavily relies on objects, not direct DB - see @67@;
- HTTP requests may be made available using Client-Server services via methods - see @49@, e.g. for rendering HTML pages generated on the fly with {\i @*Mustache@} templates- see @81@, pictures or @*PDF@ reports;
- You can little by little move your logic out of the client side code into some server services defined via interfaces, without the overhead of SOAP or WCF - see @63@; migration to @*SOA@ is the main benefit of {\i mORMot} for existing projects;
- Make your application ready to offer a RESTful interface, e.g. for consuming JSON content via AJAX or mobile clients - see @86@;
- New tables may be defined via the ORM/ODM features of {\i mORMot}, still hosted in your external SQL server - see @27@, as any previous data; in particular, mixed pure-ORM and regular-SQL requests may coexist; or {\i mORMot}'s data modeling may balance your storage among several servers (and technologies, like NoSQL);
- Sharing the same tables between legacy code SQL and {\i mORMot} ORM is possible, but to avoid consistency problems, you should better follow some rules detailed @106@;
- You may benefit from our very fast in-memory engine, a dedicated {\i @*SQLite3@}-based consolidation database or even the caching features - see @38@, shared on the server side, when performance is needed - it may help integrating some @*CQRS@ pattern ({\i Command Query Responsibility Segregation}) into your application via a @*REST@ful interface, and delegate some queries from your main database;
- If you are still using an old version of {\i Delphi}, and can't easily move up due to some third party components or existing code base, {\i mORMot} will offer all the needed features to start ORM, N-Tier and SOA, starting with a {\i Delphi} 6 edition.
{\i mORMot} implements the needed techniques for introducing what Michael Feathers calls, in his book {\i Working Effectively With Legacy Code}, a @**seam@. A seam is an area where you can start to cleave off some legacy code and begin to introduce changes. Even mocking abilities of {\i mORMot} - see @62@ - will help you in this delicate task - see @http://www.infoq.com/articles/Utilizing-Logging
Do not forget that {\i Synopse}, as a company, is able to offer dedicated audit and support for such a migration. The sooner, the better.
\page
:123 FAQ
Before you start going any further, we propose here below a simple @**FAQ@ containing the most frequent questions we received on our forums.
First of all, take a look at the {\i keyword index} available at the very beginning of this document. The underlined entries target the main article(s) about a given concept or technical term.
Feel free to give your feedback at @https://synopse.info/forum asking new questions or improving answers!
{\b Your SAD doc is too long to read through in a short period.}\line Too much documentation can kill the documentation! But you do not need to read the whole document: most of it is a detailed description of every unit, object, or class. But the first part is worth reading, otherwise you are very likely to miss some main concepts or patterns. It just takes 15-30 minutes! Also read @204@ to find out in which direction you may need to go for writing your server code. Consider the  slides available at @https://drive.google.com/folderview?id=0B0r8u-FwvxWdeVJVZnBhSEpKYkE
{\b Where should I start?}\line Take a look at the {\i Architecture principles} @40@, then download and install the sources @44@, then compile and run the {\f1\fs20 TestSQL3.dpr} program. Check about @*ORM@ @3@, @*SOA@ @63@ and @*MVC@ @108@, then test the various samples (from the {\f1\fs20 SQLite3\\Samples} folder), especially 01, 02, 04, 11, 12, 14, 17, 26, 28, 30 and the {\f1\fs20 MainDemo}.
{\b So far, I can see your {\i mORMot} fits most of the requirement, but seems only for Database Client-Server apps.}\line First of all, the framework is a {\i set of bricks}, so you can use it e.g. to build interface based services, even with no database at all. We tried to make its main features modular and uncoupled.
{\b I am not a great fan of ORM, sorry, I still like SQL and have some experience of that. Some times sophisticated SQL query is hard to change to ORM code.}\line ORM can make development much easier; but you can use e.g. interface-based services and "manual" SQL statements - in this case, you have at hand @27@ classes in {\i mORMot}, which allow very high performance and direct export to JSON.
{\b I am tempted by using an ORM, but {\i mORMot} forces you to inherit from a root {\f1\fs20 @*TSQLRecord@} type, whereas I'd like to use any kind of object.}\line We will discuss this in details @168@. Adding attributes to an existing class is tempting, but will pollute your code at the end, mixing persistence and business logic: see {\i @*Persistence Ignorance@} and {\i @*Aggregates@} @124@. The framework proposes a second level of {\i Object} mapping, allowing to persist any kind of {\i @*PODO@} ({\i Plain Old Delphi Object}), by defining @*CQRS@ services - see @167@.
{\b I would like to replace pieces of delphi-code by using mORMot and the @*DDD@-concept in a huge system, but its legacy database doesn't have integer primary keys, and {\i mORMot} ORM expects a {\f1\fs20 TID}-like field.}\line By design, such legacy tables are not compatible with {\i SQLite3} virtual tables, or our ORM - unless you add an {\f1\fs20 ID} integer additional primary key, which may not be the best idea. Some hints: write a {\i persistence service} as {\f1\fs20 interface}/{\f1\fs20 class} (as required by DDD - see @124@); uncouple persistence and @*SOA@ services (i.e. the SOA {\f1\fs20 @*TSQLRestServer@} is a {\f1\fs20 @*TSQLRestServerFullMemory@} and not a DB/ORM {\f1\fs20 @*TSQLRestServerDB@}); reuse your existing SQL statements, with {\f1\fs20 SynDB} as access layer if possible (you will have better performance, and direct @*JSON@ support); use the ORM for @*MicroService@ local persistence (with {\i SQLite3}), and/or for new tables in your legacy DB (or another storage, e.g. @*MongoDB@).
{\b Why are you not using the latest features of the compiler, like generics or class attributes?}\line Our framework does not rely on {\i generics}, but on the power of the object pascal type system: specifying a {\f1\fs20 class} or {\f1\fs20 interface} type as parameter is safe and efficient - and generics tends to blow the executable size, lower down performance (the current RTL is not very optimized, and sometimes bugged), and hide implementation details. Some methods are available for newer version of the compiler, introducing access via generics; but it was not mandatory to depend on them. We also identified, as several Java or C# gurus, that {\f1\fs20 class} attributes may sound like a good idea, but tend to {\i pollute} the code, and introduce unexpected coupling. Last but not least, those features are incompatible with older version of Delphi we would like to support, and may reduce compatibility with @*FPC@.
{\b I also notice in your SAD doc, data types are different from Delphi. You have {\f1\fs20 RawUTF8}, etc, which make me puzzled, what are they?}\line You can for sure use standard {\i Delphi} {\f1\fs20 string} types, but some more optimized types were defined: since the whole framework is @*UTF-8@ based, we defined a dedicated type, which works with all versions of {\i Delphi}, before and after {\i Delphi} 2009. By the way, just search for {\f1\fs20 RawUTF8} in the {\i keyword index} of this document, or see @32@.
{\b During my tests, my client receives non standard @*JSON@ with unquoted fields.}\line Internally, the framework uses JSON in {\i MongoDB} @*extended syntax@, i.e. fields are not quoted - this gives better performance and reduces memory and bandwidth with a {\i mORMot} client. To receive {\f1\fs20 "field":value} instead of {\f1\fs20 field:value}, just add a proper {\f1\fs20 @**User-Agent@} HTTP header to the client request (as any browser does), and the server will emit standard JSON.
{\b When I work with floating points and JSON, sometimes numerical values with more than 4 decimals are converted into JSON strings.}\line By default, {\f1\fs20 double} values are disabled in the JSON serialization, to avoid any hidden precision lost during conversion: see @194@ how to enable it.
{\b I got an access violation with SynDB ISQLDBRows.}\line You need to explicitly release the {\f1\fs20 ISQLDBRows} instance, by setting it to {\f1\fs20 nil}, {\i before} freeing the owner's connection - see @195@.
{\b @*Deadlock@ occurs with interface callbacks.}\line When working with asynchronous notifications over {\i WebSockets}, you need to ensure you won't fire directly a callback from a main method execution - see @196@ for several solutions.
{\b All the objects seem non-VCL components, meaning need code each property and remember them all well.}\line This is indeed... a feature. The framework is not @*RAD@, but fully object-oriented. Thanks to the {\i Delphi} IDE, you can access all properties description via auto-completion and/or code navigation.  We tried to make the documentation exhaustive and accurate. Then you can still use RAD for UI design, but let business be abstracted in pure code. See e.g. the {\f1\fs20 mORMotVCL.pas} unit which can publish any ORM result as {\f1\fs2 TDataSource} for your UI.
{\b I know you have joined the {\i DataSnap} performance discussion and your performance won good reputation there. If I want to use your framework to replace my old project of DataSnap, how easy will it be?}\line If you used {\i DataSnap} to build method-based services, translation into {\i mORMot} will be just a matter of code refactoring. And you will benefit of new features like {\i Interface-based services} - see @63@ - which is much more advanced than the method-based pattern, and will avoid generating the client class via a wizard, and offers additional features - see @77@ or @72@.
{\b What is the SMS? Do you know any advantage compared to JQuery or AngularJS?}\line {\i @*Smart Mobile Studio@} is an IDE and some source runtime able to develop and compile an Object-Pascal project into a {\i @*HTML 5@ / @*CSS 3@ / @*JavaScript@} {\i embedded} application, i.e. able to work stand alone with no remote server. When used with {\i mORMot} on the server side, you can use the very same object pascal language on both server and client sides, with strong typing and true @*OOP@ design. Then you feature secure authentication and JSON communication, with connected or off-line mode. Your {\i SmartPascal} client code can be generated by your {\i mORMot} server, as stated @90@. We currently focus on TMS Web Core integration, which seems a newer - and more supported - alternative.
{\b I am trying to search a substitute solution to WebSnap. Do you have any sample or doc to describe how to build a robust web Server?}\line You can indeed easily create a modern @*MVC@ / @*MVVM@ scaling @*Web Application@. Your {\i mORMot} server can easily publish its ORM / SOA business logic as {\i Model}, use {\i @*Mustache@} logic-less templates rendering - see @81@ - for {\i Views}, and defining the {\i ViewModel} / {\i Controller} as regular Delphi methods. See @108@ for more details, and discovering a sample "blog" application.
{\b Have you considered using a popular source coding host like @*Github@ or BitBucket?}\line We love to host our own source code repository, and find fossil a perfect match for our needs, with a friendly approach. But we created a parallel repository on {\i GitHub}, so that you may be able to monitor or fork our projects - see @https://github.com/synopse/mORMot \line Note that you can get a daily snapshot of our official source code repository directly from\line @https://synopse.info/files/mORMotNightlyBuild.zip
{\b Why is this framework named {\i mORMot}?}\line - Because its initial identifier was "{\i Synopse SQLite3 database framework}", which may induce a {\i SQLite3}-only library, whereas the framework is now able to connect to any database engine;\line - Because we like mountains, and those large ground rodents;\line - Because marmots do hibernate, just like our precious objects;\line - Because marmots are highly social and use loud whistles to communicate with one another, just like our applications are designed not to be isolated;\line - Because even if they eat greens, they use to fight at Spring for their realm;\line - Because it may be an acronym for "Manage Object Relational Mapping Over Territory", or whatever you may think of...
\page
:40Architecture principles
%cartoon08.png
This framework tries to implement some "best-practice" patterns, among them:
- {\i Model-View Controller} - see @10@;
- {\i Multi-@*tier@ architecture} - see @7@;
- {\i @*Test@-Driven Design} - see @12@;
- {\i @*Stateless@} @*CRUD@/@*REST@ - see @9@;
- {\i Object-Relational Mapping} - see @13@;
- {\i Object-Document Mapping} - see @82@;
- {\i @*Service@-Oriented Architecture} - see @17@.
All those points render possible any project implementation, up to complex {\i @*Domain-Driven@} design - see @54@.
\page
: General design
A general design of the {\i mORMot} architecture is shown in the following diagram:
\graph mORMotDesign3 General mORMot architecture - Client Server implementation
\ RESTful Client\RESTful Server\REST¤JSON
\RESTful Client\RESTful Server\REST¤JSON
subgraph cluster_1 {
label=" mORMot Server";
\MVC/MVVM¤Web Server\RESTful Server
\RESTful Server\Business¤rules\uses
\Business¤rules\ORM
\Services¤implementation\ORM
\Authentication¤(users, sessions)\ORM
\RESTful Server\Authentication¤(users, sessions)\requires
\RESTful Server\external tables\makes¤CRUD
\RESTful Server\in-memory tables¤JSON or binary files\makes¤CRUD
\RESTful Server\NoSQL¤engine
\RESTful Server\SQlite3¤engine\REST to SQL
\RESTful Server\Services¤implementation\runs
\SQlite3¤engine\in-memory tables¤JSON or binary files
\SQlite3¤engine\external tables
\SQlite3¤engine\SQLite3 data file\makes¤CRUD
\ORM\SQlite3¤engine
\ORM\in-memory tables¤JSON or binary files
\ORM\external tables
\ORM\NoSQL¤engine
\ORM\Redirected¤TSQLRestServer¤ORM
\ORM\Redirected¤TSQLRestClient
}
\external tables\External¤Database 1\SQL
\external tables\External¤Database 2
\NoSQL¤engine\MongoDB
subgraph cluster_2 {
"External\nDatabase 1";
}
subgraph cluster_3 {
"External\nDatabase 2";
}
subgraph cluster_4 {
"MongoDB";
}
subgraph cluster_5 {
\Redirected¤TSQLRestClient\Remote¤mORMot¤Server\REST¤JSON
}
\Desktop Browser\MVC/MVVM¤Web Server\HTTP¤HTML
\Mobile Browser\MVC/MVVM¤Web Server
subgraph cluster_6 {
label="Web Clients";
"Desktop Browser";
"Mobile Browser";
}
subgraph cluster_7 {
label="REST Client (FPC/FMX/Smart/Java/C#..)";
\ Services\ RESTful Client
\ORM methods ¤over TSQLRecord\ RESTful Client\requests
\User Interface \ORM methods ¤over TSQLRecord\asks
\User Interface \ Services
}
subgraph cluster_8 {
label="mORMot Client (Delphi)";
\Services\RESTful Client
\RESTful Client\Business rules\uses
\User Interface\Business rules\uses
\Reporting\ORM methods¤over TSQLRecord\asks
\ORM methods¤over TSQLRecord\RESTful Client\requests
\User Interface\ORM methods¤over TSQLRecord\asks
\User Interface\Services
\User Interface\Reporting\runs
\Services=ORM methods¤over TSQLRecord
}
\
;subgraph cluster_8 {
;label="Client (Java/C#/JavaScript)";
;\  Services\RESTful  Client
;\REST Resources\RESTful  Client\requests
;\User  Interface\REST Resources\uses
;\User  Interface\  Services
;}
In addition, you may use the following transversal features:
\graph mORMotDesign4 General mORMot architecture - Cross-Cutting features
subgraph cluster_9 {
label="Cross-Cutting features";
"File process\nCompression";
"Security\nCryptography";
"Remote access\nObjects cache";
"JSON\nUnicode\nUTF-8";
"Logging\nUnit Testing\nMocking";
}
\
Don't be afraid. Such a drawing may sound huge and confusing, especially when you have a RAD background, and did not work much with modern design patterns.
Following pages will detail and explain how the framework implements this architecture, and sample code is available to help you discovering the amazing {\i mORMot} realm.
In the previous diagram, you can already identify some key concepts of {\i mORMot}:
- Cross-Platform, multi clients, and multi devices;
- Can integrate to an existing code base or architecture;
- Client-Server RESTful design;
- Layered (multi-tier) implementation;
- Process can be defined via a set of Services (SOA);
- Business rules and data model are shared by Clients and Server;
- Data is mapped by objects (ORM/ODM);
- Databases can be an embedded {\i SQLite3}, one or several standard RDBMS (with auto-generated SQL), a {\i MongoDB} NoSQL engine, fast in-memory objects lists, or another {\i mORMot} server;
- Security (authentication and authorization) is integrated to all layers;
- User interface and reporting classes are available;
- You can write a MVC/MVVM AJAX or Web Application from your ORM/SOA methods;
- Based on simple and proven patterns (REST, JSON, MVC, SOLID);
- A consistent testing and debugging API is integrated;
- Optimized for both scaling and stability.
\page
: Architecture Design Process
First point is to state that you can't talk about architecture in isolation. Architecture is always driven by the actual needs of the application, not by whatever the architect read about last night and is dying to see how it works in the real world. There is no such "one architecture fits all" nor "one framework fits all" solution. Architecture is just a thinking of {\i how} you are building your own software.
In fact, software architecture is not about theory and diagrams, nor just about best practice, but about a way of implementing a working solution for your customers.
\graph ArchiIteration Architecture Iterative Process (SCRUM)
subgraph cluster_0 {
label="Customer";
\Software\Client
}
subgraph cluster_1 {
label=" Team";
\Client\Use Cases\Requirements
subgraph cluster_2 {
label="BackLog";
"Use Cases";
}
subgraph cluster_3 {
label="Design"
\Use Cases\Architecture¤Proposal
\Use Cases\Risk¤Assessment
\Use Cases\Technology¤& Models
}
subgraph cluster_4 {
label="    Dev"
\Risk¤Assessment\Tasks
\Architecture¤Proposal\Tasks
\Technology¤& Models\Tasks
}
}
\Tasks\Software\Definition¤of Done
\
This diagram presents how Architecture is part of a typical SCRUM iterative agile process. Even if some people of your company may be in charge of global software architecture, or even if your project managements follows a classic V-cycle and does not follow the agile manifesto, architecture should never be seen as a set of rules, to be applied by every and each developers. Architecture is part of the coding, but not all the coding.
Here are some ways of achieving weak design:
- Let each developer decides, from his/her own knowledge (and mood?), how to implement the use cases, with no review, implementation documentation, nor peer collaboration;
- Let each team decides, from its own knowledge (and untold internal leadership?), how to implement the use cases, with no system-wide collaboration;
- Let architecture be decided at so high level that it won't affect the actual coding style of the developers (just don't be caught);
- Let architecture be so much detailed that each code line has to follow a typical implementation pattern, therefore producing over engineered code;
- Let architecture map the existing, with some middle-term objectives at best;
- Let technology, frameworks or just-blogged ideas be used with no discrimination (do not trust the sirens of dev marketing).
Therefore, some advices:
- Collaboration is a need - no one is alone, no team is better, no manager is always right;
- Sharing is a need - between individuals, as teams, with managers;
- Stay customer and content focused;
- Long term is prepared by today's implementation;
- Be lazy, i.e. try to make tomorrow's work easier for you and your team-workers;
- They did not know it was impossible, so they did it.
Purpose of frameworks like {\i mORMot} is to provide your teams with working and integrated set of classes, so that you can focus on your product, enjoying the collaboration with other Open Source users, in order to use evolving and pertinent software architecture.
\page
:10 Model-View-Controller
The {\i Model-View-Controller} (@**MVC@) is a software architecture, currently considered an architectural pattern used in software engineering. The pattern isolates "domain logic" (the application logic for the user) from the user interface (input and presentation), permitting independent development, testing and maintenance of each (separation of concerns).
\graph MVCModel1 Model View Controller process
rankdir=LR;
\Controller\Model\Use
\Controller\View\Refresh
\Model\Controller\Notify updates
\View\Controller\Command
\
The {\b @*Model@} manages the behavior and data of the application domain, responds to requests for information about its state (usually from the view), and responds to instructions to change state (usually from the controller). In @*Event-Driven@ systems, the model notifies observers (usually views) when the information changes so that they can react - but since our ORM is @*stateless@, it does not need to handle those events - see @15@.
The {\b View} renders the model into a form suitable for interaction, typically a user interface element. Multiple views can exist for a single model for different purposes. A {\i viewport} typically has a one to one correspondence with a display surface and knows how to render to it.
The {\b Controller} receives user input and initiates a response by making calls on model objects. A controller accepts input from the user and instructs the model and viewport to perform actions based on that input.
\graph MVCModel2 Model View Controller concept
rankdir=LR;
\Model\View\indirect¤association
\Controller\Model\direct¤association
\View\Controller\indirect¤association
\Controller\View\direct¤association
\
In the framework, the {\i model} is not necessarily merely a database; the {\i model} in MVC is both the data and the business/domain logic needed to manipulate the data in the application. In our ORM, a model is implemented via a {\f1\fs20 @*TSQLModel@} class, which centralizes all {\f1\fs20 @*TSQLRecord@} inherited classes used by an application, both database-related and business-logic related.
The {\i views} can be implemented using:
- For Desktop clients, a full set of User-Interface units of the framework, which is mostly auto-generated from code - they will consume the {\i model} as reference for rendering the data;
- For Web clients, an integrated high-speed {\i @*Mustache@} rendering engine - see @81@ - is able to render HTML pages with logic-less templates, and {\i controller} methods written in Delphi - see @108@;
- For @*AJAX@ clients, the server side is easy to be reached from @*REST@ful @*JSON@ services.
The {\i controller} is mainly already implemented in our framework, within the RESTful commands, and will interact with both the associated {\i view} (e.g. for refreshing the User Interface) and {\i model} (for data handling). Some custom actions, related to the business logic, can be implemented via some custom {\f1\fs20 TSQLRecord} classes or via custom RESTful @*Service@s - see @11@.
\page
:7 Multi-tier architecture
In software engineering, multi-@**tier@ architecture (often referred to as {\i n-tier} architecture) is a client–server architecture in which the presentation, the application processing, and the data management are logically separate processes. For example, an application that uses middle-ware to service data requests between a user and a database employs multi-tier architecture. The most widespread use of multi-tier architecture is the three-tier architecture.
In practice, a typical VCL/FMX RAD application written in Delphi has a two-tier architecture:
\graph ArchiTwoTier Two-Tier Architecture - Logical View
\Application Tier\Data Tier
\
In this approach, the {\i Application Tier} mixes the UI and the logic in forms and modules.
Both @*ORM@ and @*SOA@ aspects of our @*REST@ful framework make it easy to develop using a more versatile three-tier architecture.
\graph ArchiMultiTier Multi-Tier Architecture - Logical View
\Presentation Tier\Logic Tier
\Logic Tier\Data Tier
\
The {\i Synopse mORMot Framework} follows this development pattern:
- {\i Data Tier} is either {\i @*SQLite3@} and/or an internal very fast in-memory database; most @*SQL@ queries are created on the fly, and database table layout are defined from {\i Delphi} classes; you can also use any external database, currently {\i SQLite3, @*Oracle@, @*Jet/MSAccess@, @*MS SQL@, @*Firebird@, @*DB2@, @*PostgreSQL@, @*MySQL@, @*Informix@} and {\i @*NexusDB@} SQL dialects are handled, and even @*NoSQL@ engines like {\i @*MongoDB@} can be directly used - see @27@;
- {\i Logic Tier} is performed by pure ORM aspect and SOA implementation: you write {\i Delphi} classes which are mapped by the {\i Data Tier} into the database, and you can write your business logic as Services called as {\i Delphi} {\f1\fs20 interface}, up to a {\i @*Domain-Driven@} design - see @54@ - if your project reaches some level of complexity;
- {\i Presentation Tier} is either a {\i Delphi} Client, or an @*AJAX@ application, because the framework can communicate using @*REST@ful @*JSON@ over @*HTTP@/1.1 (the {\i Delphi} Client User Interface is generated from Code, by using @*RTTI@ and structures, not as a RAD - and the Ajax applications need to be written by using your own tools and @*JavaScript@ framework, there is no "official" Ajax framework included yet).
In fact, {\i mORMot} can scales up to a {\i @*Domain-Driven@} Design four-tier architecture - see @54@ - as such:
- {\i Presentation Tier} which can be e.g. a {\i Delphi} or AJAX client;
- {\i Application Tier} which serves JSON content according to the client application;
- {\i Business Logic Tier} which centralizes all the {\i Domain} processing, shared among all applications;
- {\i Persistence/Data Tier} which can be either in-process (like {\i @*SQLite3@} or in-memory) or external (e.g. {\i Oracle, MS SQL, DB2, PostgreSQL, MySQL, Informix}...).
\graph ArchiDDDTier Domain Driven Design n-Tier Architecture - Logical View
\Presentation Tier\Application Tier
\Application Tier\Business Logic Tier
\Business Logic Tier\Data Tier
\
Note that you have to make a difference between {\i physical} and {\i logical} n-tier architecture. Most of the time, n-Tier is intended to be a {\i physical} (hardware) view, for instance a separation between the database server and the application server, placing the database on a separate machine to facilitate ease of maintenance. In {\i mORMot}, and more generally in SOA - see @17@, we deal with {\i logical} layout, with separation of layers through interfaces - see @46@ - and the underlying hardware implementation will usually not match the logical layout.
\graph ArchiDDDPhysical Domain Driven Design n-Tier Architecture - Physical View
subgraph cluster_0 {
label="Client 1 (Delphi)";
"Presentation Tier";
}
subgraph cluster_3 {
label="Client 2 (AJAX)";
"Presentation Tier ";
}
subgraph cluster_1 {
label="Application  Server";
\Application Tier\Business Logic Tier
}
\Presentation Tier \Application Tier
\Presentation Tier\Application Tier
subgraph cluster_2 {
label="  DB     Server";
\Business Logic Tier\Data Tier
}
\
In this document, we will focus on the logical way of thinking / coding, letting the physical deployment be made according to end-user expectations.
\page
:17 Service-Oriented Architecture (SOA)
@*Service@-Oriented Architecture (@**SOA@) is a flexible set of design principles used during the phases of systems development and integration in computing. A system based on a SOA will package functionality as a suite of inter-operable services that can be used within multiple, separate systems from several business domains.
A software service is a logicical representation of a repeatable activity that produce a precise result. In short, a {\i consumer} ask to a {\i producer} to act in order to produce a {\i result}. Most of the time, this invocation is free from any previous invocation (it is therefore called {\i stateless}).
The SOA implementations rely on a mesh of software services. Services comprise unassociated, {\i loosely coupled} units of functionality that have no calls to each other embedded in them. Each service implements {\i one action}, such as filling out an online application for an account, or viewing an online bank statement, or placing an online booking or airline ticket order. Rather than services embedding calls to each other in their source code, they use defined protocols that describe how services pass and parse messages using description meta-data.
\graph ArchiSOA Service Oriented Architecture - Logical View
rankdir=LR;
subgraph cluster0_ {
label="Consumers";
"Client A";
"Client B";
"Client C";
}
subgraph cluster1_ {
label="Service Bus";
\Client A\Publisher 1
\Client A\Publisher 2
\Client B\Publisher 2
\Client B\Publisher 3
\Client C\Publisher 3
}
subgraph cluster2_ {
label="Publishers";
\Publisher 1\Service 1
\Publisher 2\Service 2
\Publisher 3\Service 3
\Service 3\Publisher 2
}
\
Since most of those services are by definition @*stateless@, some kind of {\i service composition} is commonly defined to provide some kind of logical multi-tier orchestration of services. A higher level service invokes several services to work as a self-contained, stateless service; as a result, lower-level services can still be stateless, but the consumer of the higher level service is able to safely process some kind of transactional process.
\graph ArchiSOAComposition Service Oriented Architecture - Logical View of Composition
rankdir=LR;
subgraph cluster0_ {
label="Consumers";
"Client A";
}
subgraph cluster1_ {
label="Application Service Bus";
\Client A\Composition¤Publisher
}
subgraph cluster2_ {
label="Application Publishers";
\Composition¤Publisher\Composition¤Service
}
subgraph cluster3_ {
label="Business Service Bus";
\Composition¤Service\Publisher 1
\Composition¤Service\Publisher 2
\Composition¤Service\Publisher 3
}
subgraph cluster4_ {
label="Business Publishers";
\Publisher 1\Service 1
\Publisher 2\Service 2
\Publisher 3\Service 3
\Service 3\Publisher 2
}
\
For more details about SOA, see @http://en.wikipedia.org/wiki/Service-oriented_architecture
SOA is mainly about {\i decoupling}.\line That is, it enables implementation independence in a variety of ways, for instance:
|%15%40%40
|\b Dependency|Desired decoupling|Decoupling technique\b0
|Platform|Hardware, Framework or Operating System should not constrain choices of the Services consumers|Standard protocols, mainly Web services (e.g. SOAP or RESTful/JSON)
|Location|Consumers may not be impacted by service hosting changes|Routing and proxies will maintain Services access
|Availability|Maintenance tasks shall be transparent|Remote access allows centralized support on Server side
|Versions|New services shall be introduced without requiring upgrades of clients|Contract marshalling can be implemented on the Server side
|%
SOA and ORM - see @13@ - do not exclude themselves. In fact, even if some software architects tend to use only one of the two features, both can coexist and furthermore complete each other, in any @*Client-Server@ application:
- ORM access could be used to access to the data with objects, that is with the native presentation of the Server or Client side ({\i Delphi}, {\i @*JavaScript@}...) - so ORM can be used to provide efficient access to the data or the business logic - this is the idea of @*CQRS@ pattern;
- SOA will provide a more advanced way of handling the business logic: with custom parameters and data types, it is possible to provide some high-level Services to the clients, hiding most of the business logic, and reducing the needed bandwidth.
In particular, SOA will help leaving the business logic on the Server side, therefore will help increasing the @7@. By reducing the back-and-forth between the Client and the Server, it will also reduce the network bandwidth, and the Server resources (it will always cost less to run the service on the Server than run the service on the Client, adding all remote connection and @*serialization@ to the needed database access). Our @*interface@-based SOA model allows the same code to run on both the client and the server side, with a much better performance on the server side, but a full interoperability of both sides.
\page
:13 Object-Relational Mapping (ORM)
In practice, @**ORM@ gives a set of methods to ease high-level objects persistence into a @*RDBMS@.
Our {\i Delphi} {\f1\fs20 class} instances are not directly usable with a relational database, which is since decades the most convenient way of persisting data. So some kind of "glue" is needed to let class properties be saved into one or several tables. You can interact with the database using its native language, aka SQL. But SQL by itself is a full programming language, with diverse flavors depending on the exact backend engine (just think about how you define a column type able to store text). So writing and maintaining your SQL statements may become a time-consuming, difficult and error-prone task.
Sometimes, there will be nothing better than a tuned SQL statement, able to aggregate and join information from several tables. But most of the time, you will need just to perform some basic operations, known as @**CRUD@ (for {\i Create Retrieve Update Delete} actions) on well identified objects: this is where ORM may give you a huge hint, since it is able to generate the SQL statements for you.
The ORM works in fact as such:
\graph mORMotORMprocess ORM Process
\object¤instance\ORM\CRUD¤operations
\ORM\SQL\mapping
\SQL\RDBMS\DB client
\ORM\object¤instance
\SQL\ORM
\RDBMS\SQL
\
The ORM core retrieve information to perform the mapping:
- Object definition via its {\f1\fs20 class} type (via @*RTTI@);
- Database model as retrieved for each database engine.
\graph mORMotORMmapping ORM mapping
\ORM\class type
\ORM\data model
\class type\object instance
\data model\RDBMS
\
Since several implementation schemes are possible, we will first discuss the pros and the cons of each one.
First, here is a diagram presenting some common implementation schemes of database access with {\i Delphi} (which maps most other languages or frameworks, including C# or Java).
\graph mORMotORM Why a Client-Server ORM
\UI Components\DataBase\RAD
\UI\Delphi classes\code mapping
\Delphi classes\DataBase \Handwritten¤SQL
\UI \ORM\MVC¤binding
\ORM\ Database \Generated¤SQL
\ UI 1\Client 1\MVC/MVVM¤binding
\ UI 2 (web)\Client 2\MVC¤binding
\Client 1\Server\Secure¤protocol¤(REST)
\Client 2\Server
\Server\ORM \Persistence¤layer
\ORM \ Database  \Generated¤SQL
\
\page
The table below is a very suggestive (but it doesn't mean wrong) {\i Resumé} of some common schemes, in the {\i Delphi} world. @*ORM@ is just one nice possibility among others.
|%17%40%44
|\b\qc Scheme|Pros|Cons\b0
|Use DB views and tables, with GUI components|- @*SQL@ is a powerful language\line - Can use high-level DB tools (UML) and RAD approach|- Business logic can't be elaborated without stored procedures\line - SQL code and stored procedures will bind you to a DB engine\line - Poor Client interaction\line - Reporting must call the DB directly\line - No Multi-tier architecture
|Map DB tables or views with {\i Delphi} classes|- Can use elaborated business logic, in {\i Delphi}\line - Separation from UI and data|- SQL code must be coded by hand and synchronized with the classes\line - Code tends to be duplicated\line - SQL code could bind you to a DB engine\line - Reports can be made from code or via DB related tools\line - Difficult to implement true Multi-tier architecture
|Use a Database ORM|- Can use very elaborated business logic, in {\i Delphi}\line - SQL code is generated (in most cases) by the ORM\line - ORM will adapt the generated SQL to the DB engine|- More abstraction needed at design time (no RAD approach)\line - In some cases, could lead to retrieve more data from DB than needed\line - Not yet a true Multi-tier architecture, because ORM is for DB access only and business logic will need to create separated classes
|Use a @*Client-Server@ ORM|- Can use very elaborated business logic, in {\i Delphi}\line - SQL code is generated (in most cases) by the ORM\line - ORM will adapt the generated SQL to the DB engine\line - Services will allow to retrieve or process only needed data\line - Server can create objects viewed by the Client as if they were DB objects, even if they are only available in memory or the result of some business logic defined in {\i Delphi}\line - Complete Multi-tier architecture|- More abstraction needed at design time (no RAD approach)
|%
Of course, you'll find out that our framework implements a Client-Server ORM, which can be down-sized to @*stand-alone@ mode if needed, but which is, thanks to its unique implementation, scalable to any complex Domain-Driven Design.
As far as we found out, looking at every language and technology around, almost no other ORM supports such a native Client-Server orientation. Usual practice is to use a @17@ for remote access to the ORM. Some projects allow remote access to an existing ORM, but they are separated projects. Our {\i mORMot} is pretty unique, in respect to its @*REST@ful Client-Server orientation, from the ground up.
If you entered the {\i Delphi} world years ago, you may be pretty fluent with the RAD approach. But you probably also discovered how difficult it is to maintain an application which mixes UI components, business logic and database queries. Today's software users have some huge ergonomic expectations about software usability: some screens with grids and buttons, mapping the database, won't definitively be appealing. Using {\i mORMot}'s ORM /SOA approach will help you focus on your business and your clients expectations, letting the framework perform most of the plumbing for you.
\page
:82 NoSQL and Object-Document Mapping (ODM)
@**SQL@ is the De-Facto standard for data manipulation
- Schema-based;
- Relational-based;
- ACID by transactions;
- Time proven and efficient;
- "Almost" standard (each DB has its own column typing system).
{\i @**NoSQL@} is a new paradigm, named as such in early 2009 (even if some database engines, like {\i Lotus Domino}, may fit the definition since decades):
- {\i NoSQL} stands for "Not Only SQL" - which is more positive than "no SQL";
- Designed to scale for the web and @*BigData@ (e.g. Amazon, Google, Facebook), e.g. via easy @*replication@ and simple API;
- Relying on no standard (for both data modeling and querying);
- A lot of diverse implementations, covering any data use - @http://nosql-database.org lists more than 150 engines.
We can identify two main families of NoSQL databases:
- {\i Graph}-oriented databases;
- {\i Aggregate}-oriented databases.
{\i Graph}-oriented databases store data by their relations / associations:
\graph NoSQLGraphDB NoSQL Graph Database
\Alice\Bob\ID: 100¤Label: knows¤Since: 2001/10/03
\Bob\Alice\ID: 101¤Label: knows¤Since: 2001/10/04
\Alice\Group\ID: 102¤Label: is_member¤Since: 2005/07/01
\Group\Alice\ID: 103¤Label: members
\Bob\Group\ID: 105¤Label: is_member¤Since: 2011/02/14
\Group\Bob\ID: 104¤Label: members
=Alice=ID: 1¤Name: Alice¤Age: 18
=Bob=ID: 2¤Name: Bob¤Age: 22
=Group=ID: 3¤Type: Group¤Name: Chess
\
Such kind of databases are very useful e.g. for developing any "social" software, which will value its data by the relations between every node. Such data model does not fit well with the relational model, whereas a {\i NoSQL} engine like {\i Neo4j} handles such kind of data natively. Note that by design, {\i Graph}-oriented databases are ACID.
But the main {\i NoSQL} database family is populated by the {\i Aggregate}-oriented databases. By {\i Aggregate}, we mean the same definition as will be used @54@ for {\i Domain Driven Design}. It is a collection of data that we interact with as a unit, which forms the boundaries for ACID operations in a given model.
In fact, {\i Aggregate}-oriented databases can be specified as three main implementation/query patterns:
- Document-based (e.g. {\i @*MongoDB@, CouchDB, RavenDB});
- Key/Value (e.g. {\i Redis, Riak, Voldemort});
- Column family (e.g. {\i Cassandra, HiBase}).
Some of them can be {\i schema-less} (meaning that the data layout is not fixed, and can evolve on the fly without re-indexing the whole database) - but column-driven bases do have a schema, or even storing plain BLOB of data (this is the purpose of Key/Value engines, which focus on storage speed and rely on the client side to process the data).
In short, RDBMS stores data per table, and need to JOIN the references to get the aggregated information:
\graph NoSQLAggregJoin SQL Aggregate via JOINed tables
rankdir=LR;
node [shape=Mrecord];
struct1 [label="ID|UserName"];
struct2 [label="ID|<f0>UserID|Phone|Email"];
struct3 [label="ID|<f0>UserID|Level|Group"];
struct2:f0 -> struct1;
struct3:f0 -> struct1;
\
Whereas {\i NoSQL} stores its aggregates as documents: the whole data is embedded in one.
\graph NoSQLAggregNoSQL NoSQL Aggregate as one document
rankdir=LR;
node [shape=Mrecord];
struct1 [label="ID|UserName|Contact.Phone|Contact.Email|Access.Level|Access.Group"];
\
Which may be represented as the following @*JSON@ - see @2@ - data:
µ{
µ  "ID": 1234,
µ  "UserName": "John Smith",
µ  "Contact": {
µ               "Phone": "123-456-789",
µ               "Email": "xyz@abc.com"
µ             },
µ  "Access": {
µ              "Level": 5,
µ              "Group": "dev"
µ            }
µ}
Such a document will fit directly with the object programming model, without the need of thinking about JOINed queries and database plumbing.
As a result, we can discuss the two data models:
- {\i Relational data Model} with highly-structured table organization, and rigidly-defined data formats and record structure;
- {\i Document data Model} as a collection of complex documents with arbitrary, nested data formats and varying "record" format.
The {\i Relational} model features {\i @**normalization@} of data, i.e. organize the fields and tables of a relational database to minimize redundancy.\line On the other hand, the {\i Document} model features {\i @**denormalization@} of data, to optimize the read performance of a database by adding redundant data or by grouping data. It also features horizontal scaling of the servers, since data can easily be balanced among several servers, without the speed penalty of performing a remote JOIN.
One of the main difficulties, when working with {\i NoSQL}, is to define how to {\i denormalize} the data, and when to store the data in {\i normalized} format.\line One good habit is to model your data depending on the most current queries you will have to perform. For instance, you may embed sub-documents which will be very likely to be requested by your application most of the time. Note that most {\i NoSQL} engines feature a {\i projection} mechanism, which allows you to return only the needed fields for a query, leaving the sub-documents on the server if you do not need them at this time. The less frequent queries may be executed over separated collections, populated e.g. with consolidated information.\line Since {\i NoSQL} databases have fewer hard-and-fast rules than their relational databases ancestors, you are more likely to tune your model, depending on your expectations. In practice, you may spend less time thinking about "how" to store the data than with a RDBMS, and are still able to {\i normalize} information later, if needed. {\i NoSQL} engines do not fear redundant information, as soon as you follow the rules of letting the client application take care of the whole data consistency (e.g. via one ORM).
As you may have stated, this {\i Document data Model} is much closer to the @*OOP@ paradigm than the classic relational scheme. Even a new family of frameworks did appear together with {\i NoSQL} adoption, named {\i Object Document Mapping} (@**ODM@), which is what @13@ was for RDBMS.
In short, both approaches have benefits, which are to be weighted.
|%50%50
|\b SQL|NoSQL\b0
|Ubiquitous SQL|Map OOP and complex types\line (e.g. arrays or nested documents)
|Easy vertical scaling|Uncoupled data: horizontal scaling
|Data size (avoid duplicates and with no schema)|Schema-less: cleaner evolution
|Data is stored once, therefore consistent|Version management (e.g. {\i CouchDB})
|Complex ACID statements|Graph storage (e.g. {\i Redis})
|@*Aggregation@ functions (depends)|@*Map/Reduce@ or Aggregation functions\line (e.g. since {\i MongoDB} 2.2)
|%
With {\i mORMot}, you can switch from a classic SQL engine into a trendy {\i MongoDB} server, just in one line of code, when initializing the data on the server side. You can switch from ORM to ODM at any time, even at runtime, e.g. for a demanding customer.
:54 Domain-Driven Design
:  Definition
@http://domaindrivendesign.org gives the somewhat "official" definition of {\i @**Domain-Driven@} design (DDD):
{\i Over the last decade or two, a philosophy has developed as an undercurrent in the object community. The premise of domain-driven design is two-fold:}
- {\i For most software projects, the primary focus should be on the domain and domain logic;}
- {\i Complex domain designs should be based on a model.}
{\i Domain-driven design is not a technology or a methodology. It is a way of thinking and a set of priorities, aimed at accelerating software projects that have to deal with complicated domains.}
Of course, this particular architecture is customizable according to the needs of each project. We simply propose following an architecture that serves as a baseline to be modified or adapted by architects according to their needs and requirements.
:  Patterns
In respect to other kinds of @7@, DDD introduces some restrictive patterns, for a cleaner design:
- Focus on the {\i @*Domain@} - i.e. a particular kind of knowledge;
- Define {\i @*Bounded context@s} within this domain;
- Create an evolving {\i @*Model@} of the domain, ready-to-be consumed by applications;
- Identify some kind of objects - called {\i @*Value objects@} or {\i @*Entity Objects@} / {\i @*Aggregates@};
- Use an {\i @*Ubiquitous Language@} in resulting model and code;
- Isolate the domain from other kind of concern (e.g. persistence should not be called from the domain layer - i.e. the domain should not be polluted by technical considerations, but rely on the {\i @*Factory@} and {\i @*Repository@} patterns);
- Publish the domain as well-defined uncoupled {\i @*Service@s};
- Integrate the domain services with existing applications or legacy code.
The following diagram is a map of the patterns presented and the relationships between them.\line It is inspired from the one included in the Eric Evans's reference book, {\i "@*Domain-Driven@ Design", Addison-Wesley}, 2004 (and updated to take in account some points appeared since).
\graph ArchiDDDSyntax Domain-Driven Design - Building Blocks
\MDD\Ubiquitous¤Language\define¤model with
\MDD\Bounded¤Contexts\identify¤scope with
\MDD\Services\process¤model with
\MDD\Entities\express¤model with
\MDD\VO\express¤model with
\MDD\Clean/Layered/¤Hexagonal¤Architecture\isolate¤domain with
\MDD\Events\express state¤changes with
\MDD\RAD\exclude
\Entities\Repositories\access with
\Entities\Aggregates\encapsulate with
\VO\Factories\instantiated by
\Entities\Factories\instantiated by
\Aggregates\Factories\instantiated by
\Aggregates\Repositories\access with
\VO\Aggregates\encapsulate with
=MDD=Model-Driven¤Design
=VO=Value Objects
\
You may recognize a lot of existing patterns you already met or implemented. What makes DDD unique is that those patterns have been organized around some clear concepts, thanks to decades of business software experiment.
:  Is DDD good for you?
{\i Domain-Driven design} is not to be used everywhere, and in every situation.
First of all, the following are prerequisite of using DDD:
- Identified and well-bounded domain (e.g. your business target should be clearly identified);
- You must have access to domain experts to establish a creative collaboration, in an iterative (may be {\i agile}) way;
- Skilled team, able to write clean code - note also that since DDD is more about code expressiveness than technology, it may not appear so "trendy" to youngest developers;
- You want your internal team to accumulate knowledge of the domain - therefore, outsourcing may be constrained to applications, not the core domain.
Then check that DDD is worth it, i.e. if:
- It helps you solving the problem area you are trying to address;
- It meets your strategic goals: DDD is to be used where you will get your business money, and make you distinctive from your competitors;
- You need to bring clarity, and need to solve inner complexity, e.g. modeling a lot of rules (you won't use DDD to build simple applications - in this case, RAD may be enough);
- Your business is exploring: your goal is identified, but you do not know how to accomplish it;
- Don't have all of these concerns, but at least one or two.
:  Introducing DDD
Perhaps DDD sounds more appealing to you now. In this case, our {\i mORMot} framework will provide all the bricks you need to implement it, focusing on your domain and letting the libraries do all the needed plumbing.\line If you identified that DDD is not to be used now, you will always find with {\i mORMot} the tools you need, ready to switch to DDD when it will be necessary.
@66@ will benefit from DDD patterns. Finding so-called @*seam@s, along with isolating your core domain, can be extremely valuable when using DDD techniques to refactor and tighten the highest value parts of your code. It is not mandatory to re-write your whole existing software with DDD patterns everywhere: once you have identified where your business strategy's core is, you can introduce DDD progressively in this area. Then, following continuous feedback, you will refine your code, adding regression tests, and isolating your domain code from end-user code.
For a technical introduction about DDD and how {\i mORMot} can help you implement this design, see @68@.
With {\i mORMot}, your software solution will never be stuck in a dead-end. You'll be able to always adapt to your customers need, and maximize your ROI.
;:Global Architecture layout
;In the following sections, the {\i Synopse mORMot Framework} library architecture is detailed as:
;\SourcePage

[SAD-Main]
SourcePath=Lib\SQLite3
IncludePath=Lib;Lib\SQLite3;Lib\SynDBDataset;Lib\CrossPlatform;Lib\SQLite3\DDD\dom;Lib\SQLite3\DDD\infra;Zeos\src
;Lib\SQLite3\Samples\MainDemo
SourceFile=TestSQL3.dpr;SynLog.pas;SynTests.pas;mORMot.pas;mORMoti18n.pas;mORMotToolBar.pas;mORMotUI.pas;mORMotUIEdit.pas;mORMotMVC.pas;mORMotUILogin.pas;mORMotReport.pas;mORMotUIOptions.pas;mORMotUIQuery.pas;mORMotService.pas;mORMotSQLite3.pas;mORMotHttpClient.pas;mORMotHttpServer.pas;SynSQLite3.pas;SynSQLite3Static.pas;SynSQLite3RegEx.pas;SynCrtSock.pas;SynCurl.pas;SynDB.pas;SynOleDB.pas;SynDBOracle.pas;SynDBSQLite3.pas;SynDBODBC.pas;SynDBDataset.pas;SynDBZeos.pas;SynDBFireDAC.pas;SynDBUniDAC.pas;SynDBBDE.pas;SynDBNexusDB.pas;SynDBVCL.pas;mORMotReport.pas;mORMotVCL.pas;mORMotDB.pas;mORMotFastCGIServer.pas;SynSM.pas;SynDBMidasVCL.pas;mORMotMidasVCL.pas;SynMongoDB.pas;SynFastWideString.pas;SynCrossPlatformJSON.pas;SynCrossPlatformREST.pas;SynCrossPlatformSpecific.pas;SynCrossPlatformTests.pas;dddInfraSettings.pas
;Samples\MainDemo\SynFile.dpr
SourceIgnoreSymbol=select,check,open,connect,send,sqlite3,mORMot,JavaScript,cypher,execute,cache
SourceIgnoreSymbolByUnit=SynCrossPlatformJSON,SynCrossPlatformREST,SynCrossPlatformSpecific,SynCrossPlatformTests
Version=1.18
TitleOffset=0
DisplayName=mORMot Framework

:Enter new territory
: Meet the mORMot
The {\i Synopse mORMot} framework consists in a huge number of units, so we will start by introducing them.
\graph mORMotSourceCodeMainUnits mORMot Source Code Main Units
node [shape="box"];
rankdir=LR;
\mORMot.pas\SynCommons.pas
\mORMot.pas\SynTests.pas
\mORMot.pas\SynLog.pas
\mORMotDB.pas\mORMot.pas
\mORMotDB.pas\SynCommons.pas
\mORMotDB.pas\SynDB.pas
\UI\mORMot.pas
\UI\SynCommons.pas
\mORMotSQlite3.pas\SynCommons.pas
\mORMotSQlite3.pas\mORMot.pas
\mORMotSQlite3.pas\SynSQLite3.pas¤SynSQLite3Static.pas
\SynSQLite3.pas¤SynSQLite3Static.pas\SynCommons.pas
\SynDB.pas\SynCommons.pas
\SynDB.pas\SynLog.pas
\SynDBSQlite3.pas\SynDB.pas
\SynDBSQlite3.pas\SynCommons.pas
\SynDBSQlite3.pas\SynSQLite3.pas¤SynSQLite3Static.pas
\SynDBOracle.pas¤SynDBODBC.pas¤SynOleDB.pas¤SynDBZEOS.pas¤SynDBDataset.pas¤SynDBFireDAC.pas¤SynDBUniDAC.pas¤SynDBNexusDB.pas¤SynDBBDE.pas¤SynDBRemote.pas\SynDB.pas
\mORMotDB.pas\SynSQLite3.pas¤SynSQLite3Static.pas
\mORMotReport.pas\SynCommons.pas
\mORMotReport.pas\SynPdf.pas
\SynPdf.pas\SynCommons.pas
\SynSMAPI.pas¤SynSM.pas\SynCommons.pas
\HTTP\SynCrtSock.pas
\HTTP\mORMot.pas
\HTTP\SynCommons.pas
\mORMotMongoDB.pas\SynMongoDB.pas
\mORMotMongoDB.pas\mORMot.pas
\mORMotMVC.pas\mORMot.pas
\mORMotMVC.pas\SynCommons.pas
\mORMotMVC.pas\SynMustache.pas
\SynMongoDB.pas\SynCommons.pas
\SynMongoDB.pas\SynLog.pas
\SynMongoDB.pas\SynCrtSock.pas
=UI=mORMotUI.pas¤mORMotToolBar.pas¤mORMoti18n.pas¤mORMotUILogin.pas¤mORMotUIEdit.pas¤mORMotUIQuery.pas¤mORMotUIOptions.pas
=HTTP=mORMotHttpClient.pas¤mORMotHttpServer.pas
\mORMotReport.pas=UI=SynSMAPI.pas¤SynSM.pas
\
;When you will finish reading this document, you won't be afraid any more! :)
\page
: Main units
The main units you have to be familiar with are the following:
|%30%70
|\b Unit name|Description\b0
|{\f1\fs20 SynCommons.pas}\line {\f1\fs20 SynLog.pas}\line {\f1\fs20 SynTests.pas}|Common types, classes and functions
|{\f1\fs20 mORMot.pas}|Main unit of the @*ORM@ / @*SOA@ framework
|{\f1\fs20 SynSQLite3.pas\line SynSQLite3Static.pas}|{\i @*SQLite3@} database engine
|{\f1\fs20 mORMotSQLite3.pas}|Bridge between {\f1\fs20 mORMot.pas} and {\f1\fs20 SynSQLite3.pas}
|{\f1\fs20 @*SynDB@.pas\line SynDB*.pas}|Direct RDBMS access classes
|{\f1\fs20 mORMotDB.pas}|ORM external {\f1\fs20 SynDB.pas} access, via {\i SQlite3} virtual tables
|{\f1\fs20 SynMongoDB.pas\line mORMotMongoDB.pas}|Direct access to a {\i @*MongoDB@} server
|{\f1\fs20 SynSM.pas}\line {\f1\fs20 SynSMAPI.pas}|{\i SpiderMonkey} JavaScript engine
|{\f1\fs20 mORMotHttpClient.pas\line mORMotHttpServer.pas\line SynCrtSock.pas}|@*REST@ful HTTP/1.1 Client and Server
|{\f1\fs20 mORMotMVC.pas\line SynMustache.pas}|@*MVC@ classes for writing @*Web Application@s
|{\f1\fs20 mORMotUI*.pas}|Grid and Forms User Interface generation
|{\f1\fs20 mORMotToolBar.pas}|ORM ToolBar User Interface generation
|{\f1\fs20 mORMotReport.pas}|Integrated Reporting engine
|%
Other units are available in the framework source code repository, but are either expected by those files above (e.g. like {\f1\fs20 SynDB*.pas} database providers), or used only optionally in end-user @*cross-platform@ client applications (e.g. the {\f1\fs20 CrossPlatform} folder).
In the following pages, the features offered by those units will be presented.\line Do not forget to take a look at all sample projects available in the {\f1\fs20 SQLite3\\Samples} sub-folders - nothing is better than some simple code to look at.
Then detailed information will be available in the second part of this document - see @44@.
:45SynCommons unit
%cartoon01.png
First of all, let us introduce some cross-cutting features, used everywhere in the {\i Synopse} source code. Even if you do not need to go deeply into the implementation details, it will help you not be disturbed with some classes and types you may encounter in the framework source, and its documentation.
It was a design choice to use some custom low-level types, classes and functions instead of calling the official {\i Delphi} RTL.\line Benefits could be:
- Cross-platform and cross-compiler support (e.g. leverage specificities, about memory model or RTTI);
- Unicode support for all versions of {\i Delphi}, even before {\i Delphi} 2009, or with @*FPC@;
- Optimized for process speed, multi-thread friendliness and re-usability;
- Sharing of most common features (e.g. for text/data processing);
- @*KISS@ and consistent design.
In order to use {\i Synopse mORMot framework}, you should better be familiar with some of those definitions.
First of all, a {\f1\fs20 @*Synopse.inc@} include file is provided, and appears in most of the framework units:
!{$I Synopse.inc} // define HASINLINE USETYPEINFO CPU32 CPU64
It will define some conditionals, helping write portable and efficient code.
In the following next paragraphs, we'll comment some main features of the lowest-level part of the framework, mainly located in @!Lib\SynCommons.pas@:
- Unicode and @*UTF-8@;
- {\f1\fs20 @*Currency@} type;
- {\i @*Dynamic array@} wrappers ({\f1\fs20 TDynArray} and {\f1\fs20 TDynArrayHashed});
- {\f1\fs20 @*TDocVariant@} custom {\f1\fs20 variant} type for dynamic schema-less {\i object} or {\i array} storage.
Other shared features available in {\f1\fs20 SynTests.pas} and {\f1\fs20 SynLog.pas} will be detailed later, i.e. @*Test@ing and @*Log@ging - see @12@.
:32 Unicode and UTF-8
Our {\i mORMot} Framework has 100% UNICODE compatibility, that is compilation under {\i Delphi} 2009 and up (including the latest available {\i Delphi} version). The code has been deeply rewritten and @*test@ed, in order to provide compatibility with the {\f1\fs20 String=UnicodeString} paradigm of these compilers.  But the code will also handle safely Unicode for older versions, i.e. from {\i Delphi} 6 up to {\i Delphi} 2007.
From its core to its uppermost features, our framework is natively @**UTF-8@, which is the de-facto character encoding for @*JSON@, {\i @*SQLite3@}, and most supported database engines. This allows our code to offer fast streaming/parsing in a @*SAX@-like mode, avoiding any conversion between encodings from the storage layer to your business logic. We also needed to establish a secure way to use strings, in order to handle all versions of {\i Delphi} (even pre-Unicode versions, especially the {\i Delphi} 7 version we like so much), and provide compatibility with the {\i @*FreePascal@ Compiler}. This consistency allows to circumvent any RTL bug or limitation, and ease long-term support of your project.
Some string types have been defined, and used in the code for best cross-compiler efficiency:
- {\f1\fs20 @**RawUTF8@} is used for every internal data usage, since both {\i SQLite3} and JSON do expect UTF-8 encoding;
- {\f1\fs20 WinAnsiString} where {\i WinAnsi}-encoded {\f1\fs20 AnsiString} (code page 1252) are needed;
- Generic {\f1\fs20 string} for {\i @*i18n@} (e.g. in unit {\f1\fs20 mORMoti18n}), i.e. text ready to be used within the VCL, as either {\f1\fs20 AnsiString} (for {\i Delphi} 2 to 2007) or {\f1\fs20 UnicodeString} (for {\i Delphi} 2009 and later);
- {\f1\fs20 RawUnicode} in some technical places (e.g. direct Win32 *W() API call in {\i Delphi} 7) - note: this type is NOT compatible with {\i Delphi} 2009 and later {\f1\fs20 UnicodeString};
- {\f1\fs20 @**RawByteString@} for byte storage (e.g. for {\f1\fs20 FileFromString()} function);
- {\f1\fs20 @**SynUnicode@} is the fastest available Unicode {\i native} string type, depending on the compiler used (i.e. {\f1\fs20 @**WideString@} before {\i Delphi} 2009, and {\f1\fs20 UnicodeString} since);
- Some special conversion functions to be used for {\i Delphi} 2009+ {\f1 UnicodeString} (defined inside {\f1\fs20 \{$ifdef UNICODE\}...\{$endif\}} blocks);
- Never use {\f1\fs20 AnsiString} directly, but one of the types above.
Note that {\f1\fs20 RawUTF8} is the preferred {\f1\fs20 string} type to be used in our framework when defining textual properties in a {\f1\fs20 @*TSQLRecord@} and for all internal data processing. It is only when you're reaching the User Interface layer that you may convert explicitly the {\f1\fs20 RawUTF8} content into the generic VCL {\f1\fs20 string} type, using either the {\f1\fs20 Language. UTF8ToString} method (from {\f1\fs20 mORMoti18n.pas} unit) or the following function from {\f1\fs20 SynCommons.pas}:
!/// convert any UTF-8 encoded String into a generic VCL Text
!// - it's prefered to use TLanguageFile.UTF8ToString() in mORMoti18n.pas,
!// which will handle full i18n of your application
!// - it will work as is with Delphi 2009+ (direct unicode conversion)
!// - under older version of Delphi (no unicode), it will use the
!// current RTL codepage, as with WideString conversion (but without slow
!// WideString usage)
!function UTF8ToString(const Text: RawUTF8): string;
Of course, the {\f1\fs20 StringToUTF8} method or function are available to send back some text to the @*ORM@ layer.\line A lot of dedicated conversion functions (including to/from numerical values) are included in {\f1\fs20 SynCommons.pas}. Those were optimized for speed and multi-thread capabilities, and to avoid implicit conversions involving a temporary {\f1\fs20 string} variable.
Warning during the compilation process are not allowed, especially under Unicode version of {\i Delphi} (e.g. {\i Delphi} 2010): all string conversion from the types above are made explicitly in the framework's code, to avoid any unattended data loss.
If you are using older version of Delphi, and have an existing code base involving a lot of {\f1\fs20 @*WideString@} variables, you may take a look at the {\f1\fs20 @*SynFastWideString.pas@} unit. Adding this unit in the top of your {\f1\fs20 .dpr} uses clauses will let all {\f1\fs20 WideString} process use the Delphi heap and its very efficient {\i @*FastMM4@} memory manager, instead of the much slower BSTR Windows API. Performance gain can be more than 50 times, if your existing code uses a lot of {\f1\fs20 WideString} variables. Note that using this unit will break the compatibility with BSTR/COM/OLE kind of string, so is not to be used with COM objects. In all cases, if you need {\i Unicode} support with older versions of Delphi, consider using our {\f1\fs20 RawUTF8} type instead, which is much better integrated with our framework, and has less overhead.
:33 Currency handling
Faster and safer way of comparing two {\f1\fs20 @*currency@} values is certainly to map the variables to their internal {\f1\fs20 Int64} binary representation, as such:
!function CompCurrency(var A,B: currency): Int64;
!var A64: Int64 absolute A;
!    B64: Int64 absolute B;
!begin
!  result := A64-B64;
!end;
This will avoid any rounding error during comparison (working with *10000 integer values), and is likely to be faster than the default implementation, which uses the FPU (or SSE2 under {\i x64} architecture) instructions.
Some direct {\f1\fs20 currency} processing is available in the {\f1\fs20 SynCommons.pas} unit. It will by-pass the FPU use, and is therefore very fast.\line There are some functions using the {\f1\fs20 Int64} binary representation (accessible either as {\f1\fs20 PInt64(@aCurrencyVar)^} or the {\f1\fs20 absolute} syntax):
- {\f1\fs20 function Curr64ToString(Value: Int64): string;}
- {\f1\fs20 function StrToCurr64(P: PUTF8Char): Int64;}
- {\f1\fs20 function Curr64ToStr(Value: Int64): RawUTF8;}
- {\f1\fs20 function Curr64ToPChar(Value: Int64; Dest: PUTF8Char): PtrInt;}
- {\f1\fs20 function StrCurr64(P: PAnsiChar; const Value: Int64): PAnsiChar;}
Using those functions can be {\i much} faster for textual conversion than using the standard {\f1\fs20 FloatToText()} implementation. They are validated with provided regression tests.
Of course, in normal code, it is certainly not worth using the {\f1\fs20 Int64} binary representation of {\f1\fs20 currency}, but rely on the default compiler/RTL implementation. In all cases, having optimized functions was a need for both speed and accuracy of our ORM data processing, and also for @27@.
Note that we discovered some issue in the FPC compiler, when {\f1\fs20 currency} is used when compiling from {\i x64-win64}: {\f1\fs20 currency} values comparison may be wrongly implemented using {\i x87} registers: we found out that using a {\i i386-win32} FPC compiler is a safer approach, even targetting {\i x64-win64} - at least for the trunk in 2019/11.
\page
:48 TDynArray dynamic array wrapper
Version 1.13 of the {\f1\fs20 SynCommons.pas} unit introduced two kinds of wrapper:
- Low-level @*RTTI@ functions for handling record types: {\f1\fs20 RecordEquals, RecordSave, RecordSaveLength, RecordLoad};
- {\f1\fs20 TDynArray} and {\f1\fs20 TDynArrayHashed} objects, which are wrappers around any {\i @*dynamic array@}.
With {\f1\fs20 TDynArray}, you can access any {\i dynamic array} (like {\f1\fs20 TIntegerDynArray = array of integer}) using {\f1\fs20 TList}-like properties and methods, e.g. {\f1\fs20 Count, Add, Insert, Delete, Clear, IndexOf, Find, Sort} and some new methods like {\f1\fs20 LoadFromStream, SaveToStream, LoadFrom, SaveTo, Slice, Reverse,} and {\f1\fs20 AddArray}. It includes e.g. fast binary @*serialization@ of any {\i dynamic array}, even containing strings or records - a {\f1\fs20 CreateOrderedIndex} method is also available to create individual index according to the {\i dynamic array} content. You can also serialize the array content into @*JSON@, if you wish.
One benefit of {\i dynamic arrays} is that they are reference-counted, so they do not need any {\f1\fs20 Create/try..finally...Free} code, and are well handled by the {\i Delphi} compiler. For performance-critical tasks, dynamic array access is very optimized, since its whole content will be allocated at once, therefore reducing the memory fragmentation and being much more CPU cache friendly.
{\i Dynamic arrays} are no replacement to a {\f1\fs20 @*TCollection@} nor a {\f1\fs20 TList} (which are the standard and efficient way of storing class instances, and are also handled as @*published properties@ since revision 1.13 of the framework), but they are very handy way of having a list of content or a dictionary at hand, with no previous class nor properties definition.
You can look at them like Python's list, tuples (via records handling) and dictionaries (via {\f1\fs20 Find} method, especially with the dedicated {\f1\fs20 TDynArrayHashed} wrapper), in pure {\i Delphi}. Our new methods (about searching and serialization) allow most usage of those script-level structures in your {\i Delphi} code.
In order to handle {\i dynamic arrays} in our @*ORM@, some @*RTTI@-based structure were designed for this task. Since {\i dynamic array of records} should be necessary, some low-level fast access to the record content, using the common RTTI, has also been implemented (much faster than the "new" @*enhanced RTTI@ available since {\i Delphi} 2010).
:  TList-like properties
Here is how you can have method-driven access to the {\i dynamic array}:
!type
!   TGroup: array of integer;
!var
!   Group: TGroup;
!   GroupA: TDynArray;
!   i, v: integer;
!begin
!  GroupA.Init(TypeInfo(TGroup),Group); // associate GroupA with Group
!  for i := 0 to 1000 do
!  begin
!    v := i+1000; // need argument passed as a const variable
!    GroupA.Add(v);
!  end;
!  v := 1500;
!  if GroupA.IndexOf(v)<0 then // search by content
!    ShowMessage('Error: 1500 not found!');
!  for i := GroupA.Count-1 downto 0 do
!    if i and 3=0 then
!      GroupA.Delete(i); // delete integer at index i
!end;
This {\f1\fs20 TDynArray} wrapper will work also with {\f1\fs20 array of string} or {\f1\fs20 array of record}...
Records need only to be packed and have only not reference counted fields ({\f1\fs20 byte, integer, double}...) or {\f1\fs20 string} or {\f1\fs20 variant} reference-counted fields (there is no support of nested {\f1\fs20 Interface} yet). {\f1\fs20 TDynArray} is able to handle {\f1\fs20 record} within {\f1\fs20 record}, and even {\i dynamic arrays} within {\f1\fs20 record}.
Yes, you read well: it will handle a {\i dynamic array} of {\f1\fs20 record}, in which you can put some {\f1\fs20 string} or whatever data you need.
The {\f1\fs20 IndexOf()} method will search by content. That is e.g. for an {\f1\fs20 array of record}, all record fields content (including {\f1\fs20 string} properties) must match.
Note that {\f1\fs20 TDynArray} is just a wrapper around an existing {\i dynamic array} variable. In the code above, {\f1\fs20 Add} and {\f1\fs20 Delete} methods are modifying the content of the {\f1\fs20 Group} variable. You can therefore initialize a {\f1\fs20 TDynArray} wrapper on need, to access more efficiently any native {\i Delphi} {\i dynamic array}. {\f1\fs20 TDynArray} doesn't contain any data: the elements are stored in the {\i dynamic array} variable, not in the {\f1\fs20 TDynArray} instance.
:  Enhanced features
Some methods were defined in the {\f1\fs20 TDynArray} wrapper, which are not available in a plain {\f1\fs20 TList} - with those methods, we come closer to some native generics implementation:
- Now you can save and load a {\i dynamic array} content to or from a stream or a string (using {\f1\fs20 LoadFromStream/SaveToStream} or {\f1\fs20 LoadFrom/SaveTo} methods) - it will use a proprietary but very fast binary stream layout;
- And you can sort the {\i dynamic array} content by two means: either {\i in-place} (i.e. the array elements content is exchanged - use the {\f1\fs20 Sort} method in this case) or via an external integer {\i index look-up array} (using the {\f1\fs20 CreateOrderedIndex} method - in this case, you can have several orders to the same data);
- You can specify any custom comparison function, and there is a new {\f1\fs20 Find} method will can use fast binary search if available.
Here is how those new methods work:
!var
!  Test: RawByteString;
!...
!  Test := GroupA.SaveTo;
!  GroupA.Clear;
!  GroupA.LoadFrom(Test);
!  GroupA.Compare := SortDynArrayInteger;
!  GroupA.Sort;
!  for i := 1 to GroupA.Count-1 do
!    if Group[i]<Group[i-1] then
!      ShowMessage('Error: unsorted!');
!  v := 1500;
!  if GroupA.Find(v)<0 then // fast binary search
!    ShowMessage('Error: 1500 not found!');
Some unique methods like {\f1\fs20 Slice, Reverse} or {\f1\fs20 AddArray} are also available, and mimic well-known Python methods.
Still closer to the generic paradigm, working for {\i Delphi} 6 up to the latest available {\i Delphi} version, without the need of the slow enhanced RTTI, nor the executable size overhead and compilation issues of generics...
:  Capacity handling via an external Count
One common speed issue with the default usage of {\f1\fs20 TDynArray} is that the internal memory buffer is reallocated when you change its length, just like a regular {\i Delphi} {\i dynamic array}.
That is, whenever you call {\f1\fs20 Add} or {\f1\fs20 Delete} methods, an internal call to {\f1\fs20 SetLength(DynArrayVariable)} is performed. This could be slow, because it always executes some extra code, including a call to {\f1\fs20 ReallocMem}.
In order not to suffer for this, you can define an external {\i Count} value, as an {\f1\fs20 Integer} variable.
In this case, the {\f1\fs20 Length(DynArrayVariable)} will be the memory capacity of the {\i dynamic array}, and the exact number of stored item will be available from this {\i Count} variable. A {\f1\fs20 Count} property is exposed by {\f1\fs20 TDynArray}, and will always reflect the number of items stored in the {\i dynamic array}. It will point either to the external {\f1\fs20 Count} variable, if defined; or it will reflect the {\f1\fs20 Length(DynArrayVariable)}, just as usual. A {\f1\fs20 Capacity} property is also exposed by {\f1\fs20 TDynArray}, and will reflect the capacity of the {\i dynamic array}: in case of an external {\i Count} variable, it will reflect {\f1\fs20 Length(DynArrayVariable)}.
As a result, adding or deleting items could be much faster.
!var
!   Group: TIntegerDynArray;
!   GroupA: TDynArray;
!   GroupCount, i, v: integer;
!begin
!  GroupA.Init(TypeInfo(TGroup),Group,@GroupCount);
!  GroupA.Capacity := 1023; // reserver memory
!  for i := 0 to 1000 do
!  begin
!    v := i+1000; // need argument passed as a const variable
!    GroupA.Add(v); // faster than with no external GroupCount variable
!  end;
!  Check(GroupA.Count=1001);
!  Check(GroupA.Capacity=1023);
!  Check(GroupA.Capacity=length(Group));
:  JSON serialization
The {\f1\fs20 TDynArray} wrapper features some native @*JSON@ @*serialization@ abilities: {\f1\fs20 TTextWriter. AddDynArrayJSON} and {\f1\fs20 TDynArray. LoadFromJSON} methods are available for @*UTF-8@ JSON serialization of {\i dynamic arrays}.
See @53@ for all details about this unique feature.
:  Daily use
The {\f1\fs20 TTestLowLevelCommon._TDynArray} and {\f1\fs20 _TDynArrayHashed} methods implement the automated unitary tests associated with these wrappers.
You'll find out there samples of {\i dynamic array} handling and more advanced features, with various kind of data (from plain {\f1\fs20 TIntegeryDynArray} to records within records).
The {\f1\fs20 TDynArrayHashed} wrapper allow implementation of a dictionary using a {\i dynamic array} of record. For instance, the @*prepared@ statement cache is handling by the following code in @!Lib\SynSQLite3.pas@:
!  TSQLStatementCache = record
!    StatementSQL: RawUTF8;
!    Statement: TSQLRequest;
!  end;
!  TSQLStatementCacheDynArray = array of TSQLStatementCache;
!
!  TSQLStatementCached = object
!!    Cache: TSQLStatementCacheDynArray;
!!    Count: integer;
!!    Caches: TDynArrayHashed;
!    DB: TSQLite3DB;
!    procedure Init(aDB: TSQLite3DB);
!    function Prepare(const GenericSQL: RaWUTF8): PSQLRequest;
!    procedure ReleaseAllDBStatements;
!  end;
Those definitions will prepare a {\i dynamic array} storing a {\f1\fs20 TSQLRequest} and {\i SQL statement} association, with an external {\f1\fs20 Count} variable, for better speed.
It will be used as such in {\f1\fs20 @*TSQLRestServerDB@}:
!constructor TSQLRestServerDB.Create(aModel: TSQLModel; aDB: TSQLDataBase);
!begin
!  fStatementCache.Init(aDB);
! (...)
The wrapper will be initialized in the object constructor:
!procedure TSQLStatementCached.Init(aDB: TSQLite3DB);
!begin
!!  Caches.Init(TypeInfo(TSQLStatementCacheDynArray),Cache,nil,nil,nil,@Count);
!  DB := aDB;
!end;
The {\f1\fs20 TDynArrayHashed.Init} method will recognize that the first {\f1\fs20 TSQLStatementCache} field is a {\f1\fs20 @*RawUTF8@}, so will set by default an {\f1\fs20 AnsiString} hashing of this first field (we could specify a custom hash function or content hashing by overriding the default {\f1\fs20 nil} parameters to some custom functions).
So we can specify directly a {\f1\fs20 GenericSQL} variable as the first parameter of {\f1\fs20 FindHashedForAdding}, since this method will only access to the first field {\f1\fs20 RawUTF8} content, and won't handle the whole record content. In fact, the {\f1\fs20 FindHashedForAdding} method will be used to make all the hashing, search, and new item adding if necessary - just in one step. Note that this method only prepare for adding, and code needs to explicitly set the {\f1\fs20 StatementSQL} content in case of an item creation:
!function TSQLStatementCached.Prepare(const GenericSQL: RaWUTF8): PSQLRequest;
!var added: boolean;
!begin
!!  with Cache[Caches.FindHashedForAdding(GenericSQL,added)] do begin
!    if added then begin
!!      StatementSQL := GenericSQL; // need explicit set the content
!      Statement.Prepare(DB,GenericSQL);
!    end else begin
!      Statement.Reset;
!      Statement.BindReset;
!    end;
!    result := @Statement;
!  end;
!end;
The latest method of {\f1\fs20 TSQLStatementCached} will just loop for each statement, and close them: you can note that this code uses the dynamic array just as usual:
!procedure TSQLStatementCached.ReleaseAllDBStatements;
!var i: integer;
!begin
!  for i := 0 to Count-1 do
!    Cache[i].Statement.Close; // close prepared statement
!  Caches.Clear; // same as SetLength(Cache,0) + Count := 0
!end;
The resulting code is definitively quick to execute, and easy to read/maintain.
:199  TDynArrayHashed
If your purpose is to access a {\i dynamic array} using one of its fields as key, consider using {\f1\fs20 @**TDynArrayHashed@}. This wrapper, inheriting from {\f1\fs20 @*TDynArray@}, will store an hashed index of one field of the dynamic array {\f1\fs20 record}, for very efficient lookup. For a few dozen entries, it won't change the performance, but once you reach thousands of items, an index will be much faster - almost O(1) instead of O(n).
In respect to {\f1\fs20 TDynArray}, {\f1\fs20 TDynArrayHashed} instance lifetime should be consistent with the dynamic array itself, to ensure the hashed index is properly populated. You should also ensure that the dynamic array content is modified mainly via the {\f1\fs20 TDynArrayHashed.FindHashedForAdding} {\f1\fs20 TDynArrayHashed.FindHashedAndUpdate} and {\f1\fs20 TDynArrayHashed.FindHashedAndDelete} methods, or explicitly call {\f1\fs20 TDynArrayHashed.ReHash} when the dynamic array content has been modified.
In practice, {\f1\fs20 TDynArrayHashed.FindHashed} will be much faster than a regular {\f1\fs20 TDynArray.Find} call.
:200  TSynDictionary
One step further is available with the {\f1\fs20 @**TSynDictionary@} class. It is a thread-safe dictionary to store some values from associated keys, as two separated dynamic arrays.
Each {\f1\fs20 TSynDictionary} instance will hold and store the associated dynamic arrays - this is not the case with {\f1\fs20 @*TDynArray@} and {\f1\fs20 @*TDynArrayHashed@}, which are only {\i wrappers} around an existing dynamic array variable.
One big advantage is that access to {\f1\fs20 TSynDictionary} methods are thread-safe by design:  internally, a {\f1\fs20 TSynLock} will protect the keys, maintained by a @199@ instance, and the values by a {\f1\fs20 @*TDynArray@}. Access to/from local variables will be made via explicit copy, for perfect thread safety.
For advanced use, the {\f1\fs20 TSynDictionary} offers JSON serialization and binary storage (with optional compression), and the ability to specify a timeout period in seconds, after which any call to {\f1\fs20 TSynDictionary.DeleteDeprecated} will delete older entries - which is very convenient to cache values, with optional persistence on disk. Just like your own in-process {\i Redis/MemCached} instance.
\page
:80 TDocVariant custom variant type
With revision 1.18 of the framework, we introduced two new custom types of {\f1\fs20 @*variant@}s:
- {\f1\fs20 @*TDocVariant@} kind of {\f1\fs20 variant};
- {\f1\fs20 @*TBSONVariant@} kind of {\f1\fs20 variant}.
The second custom type (which handles {\i @*MongoDB@}-specific extensions - like {\f1\fs20 ObjectID} or other specific types like dates or binary) will be presented later, when dealing with {\i MongoDB} support in {\i mORMot}, together with the @*BSON@ kind of content. BSON / {\i MongoDB} support is implemented in the {\f1\fs20 SynMongoDB.pas} unit.
We will now focus on {\f1\fs20 TDocVariant} itself, which is a generic container of JSON-like objects or arrays. This custom variant type is implemented in {\f1\fs20 SynCommons.pas} unit, so is ready to be used everywhere in your code, even without any link to the {\i mORMot} ORM kernel, or {\i MongoDB}.
:  TDocVariant documents
{\f1\fs20 @**TDocVariant@} implements a custom variant type which can be used to store any JSON/BSON document-based content, i.e. either:
- Name/value pairs, for object-oriented documents (internally identified as {\f1\fs20 dvObject} sub-type);
- An array of values (including nested documents), for array-oriented documents (internally identified as {\f1\fs20 dvArray} sub-type);
- Any combination of the two, by nesting {\f1\fs20 TDocVariant} instances.
Here are the main features of this custom variant type:
- DOM approach of any {\i object} or {\i array} documents;
- Perfect storage for dynamic value-objects content, with a {\i schema-less} approach (as you may be used to in scripting languages like {\i Python} or {\i JavaScript});
- Allow nested documents, with no depth limitation but the available memory;
- Assignment can be either {\i per-value} (default, safest but slower when containing a lot of nested data), or {\i per-reference} (immediate reference-counted assignment);
- Very fast JSON serialization / un-serialization with support of {\i MongoDB}-like @*extended syntax@;
- Access to properties in code, via @*late-binding@ (including almost no speed penalty due to our VCL hack as detailed in @SDD-DI-2.2.3@);
- Direct access to the internal variant {\i names} and {\i values} arrays from code, by trans-typing into a {\f1\fs20 TDocVariantData record};
- Instance life-time is managed by the compiler (like any other {\f1\fs20 variant} type), without the need to use {\f1\fs20 interfaces} or explicit {\f1\fs20 try..finally} blocks;
- Optimized to use as little memory and CPU resource as possible (in contrast to most other libraries, it does not allocate one {\f1\fs20 class} instance per node, but rely on pre-allocated arrays);
- Opened to extension of any content storage - for instance, it will perfectly integrate with @*BSON@ serialization and custom {\i @*MongoDB@} types ({\i ObjectID, Decimal128, RegEx}...), to be used in conjunction with {\i MongoDB} servers;
- Perfectly integrated with our @48@ and its JSON serialization - see @53@, as with the {\f1\fs20 record} serialization - see @51@;
- Designed to work with our {\i mORMot} @*ORM@: any {\f1\fs20 @*TSQLRecord@} instance containing such {\f1\fs20 variant} custom types as published properties will be recognized by the ORM core, and work as expected with any database back-end (storing the content as JSON in a TEXT column);
- Designed to work with our {\i mORMot} @*SOA@: any {\f1\fs20 interface}-based service - see @63@ - is able to consume or publish such kind of content, as {\f1\fs20 variant} kind of parameters;
- Fully integrated with the {\i Delphi} IDE: any {\f1\fs20 variant} instance will be displayed as JSON in the IDE debugger, making it very convenient to work with.
To create instances of such {\f1\fs20 variant}, you can use some easy-to-remember functions:
- {\f1\fs20 _Obj() _ObjFast()} global functions to create a {\f1\fs20 variant} {\i object} document;
- {\f1\fs20 _Arr() _ArrFast()} global functions to create a {\f1\fs20 variant} {\i array} document;
- {\f1\fs20 _Json() _JsonFast() _JsonFmt() _JsonFastFmt()} global functions to create any {\f1\fs20 variant} {\i object} or {\i array} document from JSON, supplied either with standard or {\i @*MongoDB@}-@*extended syntax@.
You have two non excluding ways of using the {\f1\fs20 TDocVariant} storage:
- As regular {\f1\fs20 variant} variables, then using either late-binding or faster {\f1\fs20 _Safe()} to access its data;
- Directly as {\f1\fs20 TDocVariantData} variables, then later on returing a {\f1\fs20 variant} instance using {\f1\fs20 variant(aDocVariantData)}.
Note that you do not need to protect any stack-allocated {\f1\fs20 TDocVariantData} instance with a {\f1\fs20 try..finally}, since the compiler will do it for you. This {\f1\fs20 record} type has a lot of powerful methods, e.g. to apply {\i @*map/reduce@} on the content, or do advanced searchs or marshalling.
:   Variant object documents
The more straightforward is to use @*late-binding@ to set the properties of a new {\f1\fs20 TDocVariant} instance:
!var V: variant;
! ...
!  TDocVariant.New(V); // or slightly slower V := TDocVariant.New;
!  V.name := 'John';
!  V.year := 1972;
!  // now V contains {"name":"john","year":1982}
With {\f1\fs20 _Obj()}, an {\i object} {\f1\fs20 variant} instance will be initialized with data supplied two by two, as {\i Name,Value} pairs, e.g.
!var V1,V2: variant; // stored as any variant
! ...
!  V1 := _Obj(['name','John','year',1972]);
!  V2 := _Obj(['name','John','doc',_Obj(['one',1,'two',2.5])]); // with nested objects
Then you can convert those objects into JSON, by two means:
- Using the {\f1\fs20 VariantSaveJson()} function, which return directly one @*UTF-8@ content;
- Or by trans-typing the {\f1\fs20 variant} instance into a string (this will be slower, but is possible).
! writeln(VariantSaveJson(V1)); // explicit conversion into RawUTF8
! writeln(V1);                  // implicit conversion from variant into string
! // both commands will write '{"name":"john","year":1982}'
! writeln(VariantSaveJson(V2)); // explicit conversion into RawUTF8
! writeln(V2);                  // implicit conversion from variant into string
! // both commands will write '{"name":"john","doc":{"one":1,"two":2.5}}'
As a consequence, the {\i Delphi} IDE debugger is able to display such variant values as their JSON representation. That is, {\f1\fs20 V1} will be displayed as {\f1\fs20 '\{"name":"john","year":1982\}'} in the IDE debugger {\i Watch List} window, or in the {\i Evaluate/Modify} (F7) expression tool. This is pretty convenient, and much more user friendly than any class-based solution (which requires the installation of a specific design-time package in the IDE).
You can access to the object properties via @*late-binding@, with any depth of nesting objects, in your code:
! writeln('name=',V1.name,' year=',V1.year);
! // will write 'name=John year=1972'
! writeln('name=',V2.name,' doc.one=',V2.doc.one,' doc.two=',doc.two);
! // will write 'name=John doc.one=1 doc.two=2.5
! V1.name := 'Mark';       // overwrite a property value
! writeln(V1.name);        // will write 'Mark'
! V1.age := 12;            // add a property to the object
! writeln(V1.age);         // will write '12'
Note that the property names will be evaluated at runtime only, not at compile time. For instance, if you write {\f1\fs20 V1.nome} instead of {\f1\fs20 V1.name}, there will be no error at compilation, but an {\f1\fs20 EDocVariant} exception will be raised at execution (unless you set the {\f1\fs20 dvoReturnNullForUnknownProperty} option to {\f1\fs20 _Obj/_Arr/_Json/_JsonFmt} which will return a {\f1\fs20 null} variant for such undefined properties).
In addition to the property names, some pseudo-methods are available for such {\i object} {\f1\fs20 variant} instances:
!  writeln(V1._Count);  // will write 3 i.e. the number of name/value pairs in the object document
!  writeln(V1._Kind);   // will write 1 i.e. ord(dvObject)
!  for i := 0 to V2._Count-1 do
!    writeln(V2.Name(i),'=',V2.Value(i));
!  // will write to the console:
!  //  name=John
!  //  doc={"one":1,"two":2.5}
!  //  age=12
!  if V1.Exists('year') then
!    writeln(V1.year);
!  V1.Add('key','value');  // add one property to the object
The {\f1\fs20 variant} values returned by late-binding are generated as {\f1\fs20 varByRef}, so it has two benefits:
- Much better performance, even if the nested objects are created {\i per-value} (see below);
- Allow nested calls of pseudo methods, as such:
!var V: variant;
! ...
!  V := _Json('{arr:[1,2]}');
!  V.arr.Add(3);     // will work, since V.arr is returned by reference (varByRef)
!  writeln(V);       // will write '{"arr":[1,2,3]}'
!  V.arr.Delete(1);
!  writeln(V);       // will write '{"arr":[1,3]}'
You may also trans-type your {\f1\fs20 variant} instance into a {\f1\fs20 TDocVariantData record}, and access directly to its internals.\line For instance:
! TDocVariantData(V1).AddValue('comment','Nice guy');
! with TDocVariantData(V1) do             // direct transtyping
!   if Kind=dvObject then                 // direct access to the TDocVariantKind field
!   for i := 0 to Count-1 do              // direct access to the Count: integer field
!     writeln(Names[i],'=',Values[i]);    // direct access to the internal storage arrays
By definition, trans-typing via a {\f1\fs20 TDocVariantData record} is slightly faster than using late-binding.
But you must ensure that the {\f1\fs20 variant} instance is really a {\f1\fs20 TDocVariant} kind of data before transtyping e.g. by calling {\f1\fs20 _Safe(aVariant)^} function (or {\f1\fs20 DocVariantType.IsOfType(aVariant)} or {\f1\fs20 DocVariantData(aVariant)^}), which will work even for members returned as {\f1\fs20 varByRef} via late binding (e.g. {\f1\fs20 V2.doc}):
! with _Safe(V1)^ do                        // note ^ to de-reference into TDocVariantData
!   for ndx := 0 to Count-1 do              // direct access to the Count: integer field
!     writeln(Names[ndx],'=',Values[ndx]);  // direct access to the internal storage arrays
!
! writeln(V2.doc); // will write  '{"name":"john","doc":{"one":1,"two":2.5}}'
! if DocVariantType.IsOfType(V2.Doc) then // will be false, since V2.Doc is a varByRef variant
!   writeln('never run');                 // .. so TDocVariantData(V2.doc) will fail
! with DocVariantData(V2.Doc)^ do         // note ^ to de-reference into TDocVariantData
!   for ndx := 0 to Count-1 do            // direct access the TDocVariantData methods
!     writeln(Names[ndx],'=',Values[ndx]);
!  // will write to the console:
!  //  one=1
!  //  two=2.5
In practice, {\f1\fs20 _Safe(aVariant)^} may be preferred, since {\f1\fs20 DocVariantData(aVariant)^} will raise an {\f1\fs20 EDocVariant} exception if {\f1\fs20 aVariant} is not a {\f1\fs20 TDocVariant}, but {\f1\fs20 _Safe(aVariant)^} will return a "fake" void {\f1\fs20 DocVariant} instance, in which {\f1\fs20 Count=0} and {\f1\fs20 Kind=dbUndefined}.
The {\f1\fs20 TDocVariantData} type features some additional {\f1\fs20 U[] I[] B[] D[] O[] O_[] A[] A_[] _[]} properties, which could be used to have direct typed access to the data, as {\f1\fs20 RawUTF8}, {\f1\fs20 Int64}/{\f1\fs20 integer}, {\f1\fs20 Double}, or checking if the nested document is an {\f1\fs20 O[]}bject or an {\f1\fs20 A[]}rray.
You can also allocate directly the {\f1\fs20 TDocVariantData} instance on stack, if you do not need any {\f1\fs20 variant}-oriented access to the object, but just some local storage:
!var Doc1,Doc2: TDocVariantData;
! ...
!  Doc1.Init; // needed for proper initialization
!  assert(Doc1.Kind=dvUndefined);
!  Doc1.AddValue('name','John');        // add some properties
!  Doc1.AddValue('birthyear',1972);
!  assert(Doc1.Kind=dvObject);          // is now identified as an object
!  assert(Doc1.Value['name']='John');    // read access to the properties (also as varByRef)
!  assert(Doc1.Value['birthyear']=1972);
!  assert(Doc1.U['name']='John');        // slightly faster read access
!  assert(Doc1.I['birthyear']=1972);
!  writeln(Doc1.ToJSON); // will write '{"name":"John","birthyear":1972}'
!  Doc1.Value['name'] := 'Jonas';      // update one property
!  writeln(Doc1.ToJSON); // will write '{"name":"Jonas","birthyear":1972}'
!  Doc2.InitObject(['name','John','birthyear',1972],
!    aOptions+[dvoReturnNullForUnknownProperty]); // initialization from name/value pairs
!  assert(Doc2.Kind=dvObject);
!  assert(Doc2.Count=2);
!  assert(Doc2.Names[0]='name');
!  assert(Doc2.Values[0]='John');
!  writeln(Doc2.ToJSON);         // will write '{"name":"John","birthyear":1972}'
!  Doc2.Delete('name');
!  writeln(Doc2.ToJSON);         // will write '{"birthyear":1972}'
!  assert(Doc2.U['name']='');
!  assert(Doc2.I['birthyear']=1972);
!  Doc2.U['name'] := 'Paul';
!  Doc2.I['birthyear'] := 1982;
!  writeln(Doc2.ToJSON);         // will write '{"name":"Paul","birthyear":1982}'
You do not need to protect the stack-allocated {\f1\fs20 TDocVariantData} instances with a {\f1\fs20 try..finally}, since the compiler will do it for your. Take a look at all the methods and properties of {\f1\fs20 TDocVariantData}.
:   FPC restrictions
You should take note that with the {\i @*FreePascal@} compiler, calling late-binding functions with arguments (like {\f1\fs20 Add} or {\f1\fs20 Delete}) would most probably fail to work as expected.\line We have found out that the following code may trigger some random access violations:
!  doc.Add('text');
!  doc.Add(anotherdocvariant);
So you should rather access directly the underlying {\f1\fs20 TDocVariantData} instance:
!  TDocVariantData(doc).AddItem('text');
!  TDocVariantData(doc).AddItem(anotherdocvariant);
Or even better using {\f1\fs20 _Safe()}:
!  _Safe(doc)^.AddItem('text');
!  _Safe(doc)^.AddItem(anotherdocvariant);
In fact, late-binding functions arguments seem to work only for simple values (like {\f1\fs20 integer} or {\f1\fs20 double}), but not complex types (like {\f1\fs20 string} or other {\f1\fs20 TDocVariantData}), which generate some random GPF, especially when {\f1\fs20 heaptrc} paranoid memory checks are enabled.
As a result, direct access to {\f1\fs20 TJSONVariantData} instances - preferably via {\f1\fs20 _Safe()}, and not a {\f1\fs20 variant} variable, will be faster and less error-prone when using FPC.
:   Variant array documents
With {\f1\fs20 _Arr()}, an {\i array} {\f1\fs20 variant} instance will be initialized with data supplied as a list of {\i Value1,Value2,...}, e.g.
!var V1,V2: variant; // stored as any variant
! ...
!  V1 := _Arr(['John','Mark','Luke']);
!  V2 := _Obj(['name','John','array',_Arr(['one','two',2.5])]); // as nested array
Then you can convert those objects into JSON, by two means:
- Using the {\f1\fs20 VariantSaveJson()} function, which return directly one @*UTF-8@ content;
- Or by trans-typing the {\f1\fs20 variant} instance into a string (this will be slower, but is possible).
! writeln(VariantSaveJson(V1));
! writeln(V1);  // implicit conversion from variant into string
! // both commands will write '["John","Mark","Luke"]'
! writeln(VariantSaveJson(V2));
! writeln(V2);  // implicit conversion from variant into string
! // both commands will write '{"name":"john","array":["one","two",2.5]}'
As a with any {\i object} document, the {\i Delphi} IDE debugger is able to display such {\i array} {\f1\fs20 variant} values as their JSON representation.
@*Late-binding@ is also available, with a special set of pseudo-methods:
!  writeln(V1._Count); // will write 3 i.e. the number of items in the array document
!  writeln(V1._Kind);  // will write 2 i.e. ord(dvArray)
!  for i := 0 to V1._Count-1 do
!    writeln(V1.Value(i),':',V2._(i));    // Value() or _() pseudo-methods
!  // will write in the console:
!  //  John John
!  //  Mark Mark
!  //  Luke Luke
!  if V1.Exists('John') then             // Exists() pseudo-method
!    writeln('John found in array');
!  V1.Add('new item');                   // add "new item" to the array
!  V1._ := 'another new item';           // add "another new item" to the array
!  writeln(V1);          // will write '["John","Mark","Luke","new item","another new item"]'
!  V1.Delete(2);
!  V1.Delete(1);
!  writeln(V1);          // will write '["John","Luke","another new item"]'
When using @*late-binding@, the object properties or array items are retrieved as {\f1\fs20 varByRef}, so you can even run the pseudo-methods on any nested member:
!  V := _Json('["root",{"name":"Jim","year":1972}]');
!  V.Add(3.1415);
!  assert(V='["root",{"name":"Jim","year":1972},3.1415]');
!  V._(1).Delete('year');          // delete a property of the nested object
!  assert(V='["root",{"name":"Jim"},3.1415]');
!  V.Delete(1);                    // delete an item in the main array
!  assert(V='["root",3.1415]');
Of course, trans-typing into a {\f1\fs20 TDocVariantData record} is possible, and will be slightly faster than using late-binding. As usual, using {\f1\fs20 _Safe(aVariant)^} function is safer, especially when working on {\f1\fs20 varByRef} members returned via late-binding.
As with an {\i object} document, you can also allocate directly the {\f1\fs20 TDocVariantData} instance on stack, if you do not need any {\f1\fs20 variant}-oriented access to the array:
!var Doc: TDocVariantData;
! ...
!  Doc.Init; // needed for proper initialization  - see also Doc.InitArray()
!  assert(Doc.Kind=dvUndefined);      // this instance has no defined sub-type
!  Doc.AddItem('one');                // add some items to the array
!  Doc.AddItem(2);
!  assert(Doc.Kind=dvArray);          // is now identified as an array
!  assert(Doc.Value[0]='one');         // direct read access to the items
!  assert(Doc.Values[0]='one');        // with index check
!  assert(Doc.Count=2);
!  writeln(Doc.ToJSON); // will write '["one",2]'
!  Doc.Delete(0);
!  assert(Doc.Count=1);
!  writeln(Doc.ToJSON); // will write '[2]'
You could use the {\f1\fs20 A[]} property to retrieve an object property as a {\f1\fs20 TDocVariant} array, or the {\f1\fs20 A_[]} property to add a missing array property to an object, for instance:
!  Doc.Clear;  // reset the previous Doc content
!  writeln(Doc.A['test']); // will write 'null'
!  Doc.A_['test']^.AddItems([1,2]);
!  writeln(Doc.ToJSON);    // will write '{"test":[1,2]}'
!  writeln(Doc.A['test']); // will write '[1,2]'
!  Doc.A_['test']^.AddItems([3,4]);
!  writeln(Doc.ToJSON);    // will write '{"test":[1,2,3,4]}'
:   Create variant object or array documents from JSON
With {\f1\fs20 _Json()} or {\f1\fs20 _JsonFmt()}, either a {\i document} or {\i array} {\f1\fs20 variant} instance will be initialized with data supplied as JSON, e.g.
!var V1,V2,V3,V4: variant; // stored as any variant
! ...
!  V1 := _Json('{"name":"john","year":1982}'); // strict JSON syntax
!  V2 := _Json('{name:"john",year:1982}');     // with MongoDB extended syntax for names
!  V3 := _Json('{"name":?,"year":?}',[],['john',1982]);
!  V4 := _JsonFmt('{%:?,%:?}',['name','year'],['john',1982]);
!  writeln(VariantSaveJSON(V1));
!  writeln(VariantSaveJSON(V2));
!  writeln(VariantSaveJSON(V3));
!  // all commands will write '{"name":"john","year":1982}'
Of course, you can nest objects or arrays as parameters to the {\f1\fs20 _JsonFmt()} function.
The supplied JSON can be either in strict JSON syntax, or with the {\i @*MongoDB@} @*extended syntax,@ i.e. with unquoted property names. It could be pretty convenient and also less error-prone when typing in the {\i Delphi} code to forget about @*quotes@ around the property names of your JSON.
Note that {\i TDocVariant} implements an open interface for adding any custom extensions to JSON: for instance, if the {\f1\fs20 SynMongoDB.pas} unit is defined in your application, you will be able to create any {\i MongoDB} specific types in your JSON, like {\f1\fs20 ObjectID()}, {\f1\fs20 NumberDecimal(""...")} ,{\f1\fs20 new Date()} or even {\f1\fs20 /regex/option}.
As a with any {\i object} or {\i array} document, the {\i Delphi} IDE debugger is able to display such {\f1\fs20 variant} values as their JSON representation.
:   Per-value or per-reference
By default, the {\f1\fs20 variant} instance created by {\f1\fs20 _Obj() _Arr() _Json() _JsonFmt()} will use a {\i copy-@**by-value@} pattern. It means that when an instance is affected to another variable, a new {\f1\fs20 variant} document will be created, and all internal values will be copied. Just like a {\f1\fs20 record} type.
This will imply that if you modify any item of the copied variable, it won't change the original variable:
!var V1,V2: variant;
! ...
! V1 := _Obj(['name','John','year',1972]);
! V2 := V1;                // create a new variant, and copy all values
! V2.name := 'James';      // modifies V2.name, but not V1.name
! writeln(V1.name,' and ',V2.name);
! // will write 'John and James'
As a result, your code will be perfectly safe to work with, since {\f1\fs20 V1} and {\f1\fs20 V2} will be uncoupled.
But one drawback is that passing such a value may be pretty slow, for instance, when you nest objects:
!var V1,V2: variant;
! ...
! V1 := _Obj(['name','John','year',1972]);
! V2 := _Arr(['John','Mark','Luke']);
! V1.names := V2; // here the whole V2 array will be re-allocated into V1.names
Such a behavior could be pretty time and resource consuming, in case of a huge document.
All {\f1\fs20 _Obj() _Arr() _Json() _JsonFmt()} functions have an optional {\f1\fs20 TDocVariantOptions} parameter, which allows to change the behavior of the created {\f1\fs20 TDocVariant} instance, especially setting {\f1\fs20 dvoValueCopiedByReference}.
This particular option will set the {\i copy-@**by-reference@} pattern:
!var V1,V2: variant;
! ...
! V1 := _Obj(['name','John','year',1972],[dvoValueCopiedByReference]);
! V2 := V1;             // creates a reference to the V1 instance
! V2.name := 'James';   // modifies V2.name, but also V1.name
! writeln(V1.name,' and ',V2.name);
! // will write 'James and James'
You may think that this behavior is somewhat weird for a {\f1\fs20 variant} type. But if you forget about {\i per-value} objects and consider those {\f1\fs20 TDocVariant} types as a {\i Delphi} {\f1\fs20 class} instance (which is a {\i per-reference} type), without the need of having a fixed schema nor handling manually the memory, it will probably start to make sense.
Note that a set of global functions have been defined, which allows direct creation of documents with {\i per-reference} instance lifetime, named {\f1\fs20 _ObjFast() _ArrFast() _JsonFast() _JsonFmtFast()}. Those are just wrappers around the corresponding {\f1\fs20 _Obj() _Arr() _Json() _JsonFmt()} functions, with the following {\f1\fs20 JSON_OPTIONS[true]} constant passed as options parameter:
!const
!  /// some convenient TDocVariant options
!  // - JSON_OPTIONS[false] is _Json() and _JsonFmt() functions default
!  // - JSON_OPTIONS[true] are used by _JsonFast() and _JsonFastFmt() functions
!  JSON_OPTIONS: array[Boolean] of TDocVariantOptions = (
!    [dvoReturnNullForUnknownProperty],
!!    [dvoReturnNullForUnknownProperty,dvoValueCopiedByReference]);
When working with complex documents, e.g. with @*BSON@ / {\i @*MongoDB@} documents, almost all content will be created in "fast" {\i per-reference} mode.
:  Advanced TDocVariant process
:194   Number values options
By default, {\f1\fs20 TDocVariantData} will only recognize {\f1\fs20 integer}, {\f1\fs20 Int64} and {\f1\fs20 currency} - see @33@ - as number values. Any floating point value which may not be translated to/from @*JSON@ textual representation safely will be stored as a JSON string, i.e. if it does match an integer or up to 4 fixed decimals, with 64-bit precision. We stated that JSON serialization should be conservative, i.e. serializing then unserializing (or the other way round) should return the very same value; parsing JSON is a matter of (difficult) choices - see @http://seriot.ch/parsing_json.php#5 - and we choose to be paranoid and not loose information by default.
You can use the {\f1\fs20 @*_JsonFastFloat()@} wrapper or set the {\f1\fs20 @**dvoAllowDoubleValue@} option to {\f1\fs20 TDocVariantData}, so that such floating-point numbers will be recognized and stored as {\f1\fs20 @**double@}. In this case, only {\f1\fs20 varDouble} storage will be used for the {\f1\fs20 variant} values, i.e. 64-bit IEEE 754 {\f1\fs20 double} values, handling 5.0 x 10^-324 .. 1.7 x 10^308 range. With such floating-point values, you may loose precision and digits during the JSON serialization process: this is why it is not enabled by default.
Also note that some JSON engines do not support 64-bit integer numbers. For instance, {\f1\fs20 @*JavaScript@} engines handle only up to @*53-bit@ of information without precision loss (called the {\i significand} bits), due to their internal storage as a 8 bytes IEEE 754 container. In some cases, it is safest to use JSON string representation of such numbers, as is done with the {\f1\fs20 woIDAsIDstr} value of {\f1\fs20 TTextWriterWriteObjectOption} for safe serialization of {\f1\fs20 TSQLRecord.ID} ORM values.
If you want to work with high-precision floating point numbers, consider using {\f1\fs20 @*TDecimal128@} values, as implemented in {\f1\fs20 SynMongoDB.pas}, which supports 128-bit high precision decimal, as defined by the {\i IEEE 754-2008 128-bit decimal floating point} standard, and handled in {\i MongoDB} 3.4+. Their conversion to/from text - therefore to/from JSON - won't loose nor round any digit, as soon as the value fits in its 128-bit storage.
:   Object or array document creation options
As stated above, a {\f1\fs20 TDocVariantOptions} parameter enables to define the behavior of a {\f1\fs20 TDocVariant} custom type for a given instance. Please refer to the documentation of this set of options to find out the available settings. Some are related to the memory model, other to case-sensitivity of the property names, other to the behavior expected in case of non-existing property, and so on...
Note that this setting is {\i local} to the given {\f1\fs20 variant} instance.
In fact, {\f1\fs20 TDocVariant} does not force you to stick to one memory model nor a set of global options, but you can use the best pattern depending on your exact process. You can even {\i mix} the options - i.e. including some objects as properties in an object created with other options - but in this case, the initial options of the nested object will remain. So you should better use this feature with caution.
You can use the {\f1\fs20 _Unique()} global function to force a variant instance to have an unique set of options, and all nested documents to become {\i by-value}, or {\f1\fs20 _UniqueFast()} for all nested documents to become {\i by-reference}.
!  // assuming V1='{"name":"James","year":1972}' created by-reference
!  _Unique(V1);             // change options of V1 to be by-value
!  V2 := V1;                // creates a full copy of the V1 instance
!  V2.name := 'John';       // modifies V2.name, but not V1.name
!  writeln(V1.name);        // write 'James'
!  writeln(V2.name);        // write 'John'
!  V1 := _Arr(['root',V2]); // created as by-value by default, as V2 was
!  writeln(V1._Count);      // write 2
!  _UniqueFast(V1);         // change options of V1 to be by-reference
!  V2 := V1;
!  V1._(1).name := 'Jim';
!  writeln(V1);
!  writeln(V2);
!  // both commands will write '["root",{"name":"Jim","year":1972}]'
The easiest is to stick to one set of options in your code, i.e.:
- Either using the {\f1\fs20 _*()} global functions if your business code does send some {\f1\fs20 TDocVariant} instances to any other part of your logic, for further storage: in this case, the {\i by-value} pattern does make sense;
- Or using the {\f1\fs20 _*Fast()} global functions if the {\f1\fs20 TDocVariant} instances are local to a small part of your code, e.g. used as dynamic schema-less {\i Data Transfer Objects} ({\i @*DTO@}).
In all cases, be aware that, like any {\f1\fs20 class} type, the {\f1\fs20 const}, {\f1\fs20 var} and {\f1\fs20 out} specifiers of method parameters does not behave to the {\f1\fs20 TDocVariant} value, but to its reference.
:   Integration with other mORMot units
In fact, whenever a dynamic {\i schema-less} storage structure is needed, you may use a {\f1\fs20 TDocVariant} instance instead of {\f1\fs20 class} or {\f1\fs20 record} strong-typed types:
- Client-Server ORM - see @3@ - will support {\f1\fs20 TDocVariant} in any of the {\f1\fs20 TSQLRecord variant} published properties (and store them as @*JSON@ in a text column);
- Interface-based services - see @63@ - will support {\f1\fs20 TDocVariant} as {\f1\fs20 variant} parameters of any method, which make them as perfect {\i DTO};
- Since JSON support is implemented with any {\f1\fs20 TDocVariant} value from the ground up, it makes a perfect fit for working with AJAX clients, in a script-like approach;
- If you use our {\f1\fs20 SynMongoDB.pas mORMotMongoDB.pas} units to access a {\i @*MongoDB@} server, {\f1\fs20 TDocVariant} will be the native storage to create or access nested @*BSON@ arrays or objects documents - that is, it will allow proper @*ODM@ storage;
- Cross-cutting features (like logging or {\f1\fs20 record} / {\i dynamic array} enhancements) will also benefit from this {\f1\fs20 TDocVariant} custom type.
We are pretty convinced that when you will start playing with {\f1\fs20 TDocVariant}, you won't be able to live without it any more. It introduces the full power of @*late-binding@ and dynamic schema-less patterns to your application code, which can be pretty useful for prototyping or in Agile development. You do not need to use scripting engines like {\i Python} or {\i JavaScript}: {\i Delphi} is perfectly able to handle dynamic coding!
: Cross-cutting functions
:175  Iso8601 time and date
For date/time storage as text, the framework will use {\i @**ISO 8601@} encoding. Dates could be encoded as {\f1\fs20 YYYY-MM-DD} or {\f1\fs20 YYYYMMDD}, time as {\f1\fs20 hh:mm:ss} or {\f1\fs20 hhmmss}, and combined date and time representations as {\f1\fs20 <date>T<time>}, i.e. {\f1\fs20 YYYY-MM-DDThh:mm:ss} or {\f1\fs20 YYYYMMDDThhmmss}.
The {\i lexicographical order} of the representation thus corresponds to chronological order, except for date representations involving negative years. This allows dates to be naturally sorted by, for example, file systems, or grid lists.
:   TDateTime and TDateTimeMS
In addition to the default {\f1\fs20 @**TDateTime@} type, which will be serialized with a second resolution, you may use {\f1\fs20 @**TDateTimeMS@}, which will include the milliseconds, i.e. {\f1\fs20 YYYY-MM-DDThh:mm:ss.sss} or {\f1\fs20 YYYYMMDDThhmmss.sss}:
!type
!  TDateTimeMS = type TDateTime;
This {\f1\fs20 TDateTimeMS} type is handled both during {\f1\fs20 record} - see @51@ - and dynamic array - see @53@ - JSON serialization, and by the framework {\f1\fs20 @*ORM@}.
:   TTimeLog and TTimeLogBits
The {\f1\fs20 SynCommons.pas} unit defines a {\f1\fs20 @**TTimeLog@} type, and some functions able to convert to/from regular {\f1\fs20 TDateTime} values:
!type
!  TTimeLog = type Int64;
This integer storage is encoded as a series of bits, which will map the {\f1\fs20 @**TTimeLogBits@} record type, as defined in {\f1\fs20 SynCommons.pas} unit.
The resolution of such values is one second. In fact, it uses internally for computation an abstract "year" of 16 months of 32 days of 32 hours of 64 minutes of 64 seconds.\line As a consequence, any date/time information can be retrieved from its internal bit-level representation:
- 0..5 bits will map {\i seconds},
- 6..11 bits will map {\i minutes},
- 12..16 bits will map {\i hours},
- 17..21 bits will map {\i days} (minus one),
- 22..25 bits will map {\i months} (minus one),
- 26..40 bits will map {\i years}.
The {\i ISO 8601} standard allows millisecond resolution, encoded as {\f1\fs20 hh:mm:ss.sss} or {\f1\fs20 hhmmss.sss}. Our {\f1\fs20 TTimeLog}/{\f1\fs20 TTimeLogBits} integer encoding uses a second time resolution, and a 64-bit integer storage, so is not able to handle such precision. You could use {\f1\fs20 @*TDateTimeMS@} or {\f1\fs20 @*TUnixMSTime@} values instead, if milliseconds are required.
Note that since {\f1\fs20 TTimeLog} type is bit-oriented, you can't just use {\i add} or {\i subtract} two {\f1\fs20 TTimeLog} values when doing such date/time computation: use a {\f1\fs20 TDateTime} temporary conversion in such case. See for instance how the {\f1\fs20 TSQLRest.ServerTimestamp} property is computed:
!function TSQLRest.GetServerTimestamp: TTimeLog;
!begin
!  PTimeLogBits(@result)^.From(Now+fServerTimestampOffset);
!end;
!
!procedure TSQLRest.SetServerTimestamp(const Value: TTimeLog);
!begin
!  fServerTimestampOffset := PTimeLogBits(@Value)^.ToDateTime-Now;
!end;
But if you simply want to {\i compare} {\f1\fs20 TTimeLog} kind of date/time, it is safe to directly compare their {\f1\fs20 Int64} underlying value, since timestamps will be stored in increasing order, with a resolution of one second.
Due to compiler limitation in older versions of {\i Delphi}, direct typecast of a {\f1\fs20 TTimeLog} or {\f1\fs20 Int64} variable into a {\f1\fs20 TTimeLogBits} record (as with {\f1\fs20 TTimeLogBits(aTimeLog).ToDateTime}) could lead to an internal compiler error. In order to circumvent this bug, you will have to use a {\f1\fs20 pointer} typecast, e.g. as in {\f1\fs20 TimeLogBits(@Value)^.ToDateTime} above.\line But in most case, you should better use the following functions to manage such timestamps:
! function TimeLogNow: TTimeLog;
! function TimeLogNowUTC: TTimeLog;
! function TimeLogFromDateTime(DateTime: TDateTime): TTimeLog;
! function TimeLogToDateTime(const Timestamp: TTimeLog): TDateTime; overload;
! function Iso8601ToTimeLog(const S: RawByteString): TTimeLog;
See @174@ for additional information about this {\f1\fs20 TTimeLog} storage, and how it is handled by the framework @*ORM@, via the additional {\f1\fs20 @*TModTime@} and {\f1\fs20 @*TCreateTime@} types.
:   TUnixTime and TUnixMSTime
You may consider the {\f1\fs20 @**TUnixTime@} type, which holds a 64-bit encoded number of {\i seconds} since the Unix Epoch, i.e. 1970-01-01 00:00:00 UTC:
!type
!  TUnixTime = type Int64;
You can convert such values:
- to/from {\f1\fs20 TTimeLog} values using {\f1\fs20 TTimeLogBits.ToUnixTime} and {\f1\fs20 TTimeLogBits.FromUnixTime} methods;
- to/from {\f1\fs20 TDateTime} values using {\f1\fs20 UnixTimeToDateTime}/{\f1\fs20 DateTimeToUnixTime} functions;
- using {\f1\fs20 UnixTimeUTC} to return the current timestamp, calling very fast OS API.
An alternative {\f1\fs20 @*TUnixMSTime@} type is also available, which stores the date/time as a 64-bit encoded number of {\i milliseconds} since the Unix Epoch, i.e. 1970-01-01 00:00:00 UTC. Milliseconds resolution may be handy in some cases, especially when {\f1\fs20 TTimeLog} second resolution is not enough, and you want a more standard encoding than Delphi's {\f1\fs20 TDateTime}.
You may consider using {\f1\fs20 TUnixTime} and {\f1\fs20 TUnixMSTime} especially if the timestamp is likely to be handled by third-party clients following this C/C#/Java/JavaScript encoding. In the Delphi world, {\f1\fs20 TDateTime}, {\f1\fs20 TDateTimeMS} or {\f1\fs20 TTimeLog} types could be preferred.
:176  Time Zones
One common problem when handling dates and times, is that common time is shown and entered as {\i local}, whereas the computer should better use non-geographic information - especially on a Client-Server architecture, where both ends may not be on the same physical region.
A {\i @**time zone@} is a region that observes a uniform standard time for legal, commercial, and social purposes. Time zones tend to follow the boundaries of countries and their subdivisions because it is convenient for areas in close commercial or other communication to keep the same time. Most of the time zones on land are offset from {\i Coordinated Universal Time} (@**UTC@) by a whole number of hours, or minutes. Even worse, some countries use daylight saving time for part of the year, typically by changing clocks by an hour, twice every year.
The main rule is that any date and time stored should be stored in UTC, or with an explicit Zone identifier (i.e. an explicit offset to the UTC value). Our framework expects this behavior: every date/time value stored and handled by the ORM, SOA, or any other part of it, is expected to be UTC-encoded. At presentation layer (e.g. the User Interface), conversion to/from local times should take place, so that the end-user is provided with friendly clock-wall compatible timing.
As you may guess, handling time zones is a complex task, which should be managed by the Operating System itself. Since this cultural material is constantly involving, it is updated as part of the OS.
In practice, current local time could be converted from UTC from the current system-wide time zone. One of the only parameters you have to set when installing an Operating System is to pickup the keyboard layout... and the current time zone to be used. But in a client-server environment, you may have to manage several time zones on the server side: so you can't rely on this global setting.
One sad - but predictable - disappointment is that there is no common way of encoding time zone information. Under {\i Windows}, the registry contains a list of time zones, and the associated time bias data. Most POSIX systems (including {\i Linux} and Mac OSX) do rely on the IANA database, also called {\f1\fs20 tzdata} - you may have noticed that this particular package is often updated with your system. Both zone identifiers do not map, so our framework needed something to be shared on all systems.
The {\f1\fs20 SynCommons.pas} unit features the {\f1\fs20 @**TSynTimeZone@} class, which is able to retrieve the information from the {\i Windows} registry into memory via {\f1\fs20 TSynTimeZone.LoadFromRegistry}, or into a compressed file via {\f1\fs20 TSynTimeZone.SaveToFile}. Later on, this file could be reloaded on any system, including any {\i Linux} flavor, via {\f1\fs20 TSynTimeZone.LoadFromFile}, and returns the very same results. The compressed file is pretty small, thanks to its optimized layout, and use of our {\f1\fs20 SynLZ} compression algorithm: the full information is stored in a 7 KB file - the same flattened information as JSON is around 130 KB, and you may compare with the official @http://www.iana.org content, which weighted as a 280KB {\f1\fs20 tar.gz}... Of course, {\f1\fs20 tzdata} stores potentially a lot more information than we need.
In practice, you may use {\f1\fs20 TSynTimeZone.Default}, which will return an instance read from the current version of the registry under {\i Windows}, and will attempt to load the information named after the executable file name (appended as a {\f1\fs20 .tz} extension) on other Operating Systems.\line You may therefore write:
! aLocalTime := TSynTimeZone.Default.NowToLocal(aTimeZoneID);
Similarly, you may use {\f1\fs20 TSynTimeZone.UtcToLocal} or {\f1\fs20 TSynTimeZone.LocalToUtc} methods, with the proper {\f1\fs20 TZ} identifier.
You will have to create the needed {\f1\fs20 .tz} compressed file under a Windows machine, then provide this file together with any {\i Linux} server executable, in its very same folder. On a @*Cloud@-like system, you may store this information in a centralized server, e.g. via a dedicated service - see @63@ - generated from a single reference {\i Windows} system via {\f1\fs20 TSynTimeZone.SaveToBuffer}, and later on use {\f1\fs20 TSynTimeZone.LoadFromBuffer} to decode it from all your cloud nodes. The main benefit is that the time information will stay consistent whatever system it runs on, as you may expect.
Your User Interface could retrieve the IDs and ready to be displayed text from {\f1\fs20 TSynTimeZone.Ids} and {\f1\fs20 TSynTimeZone.Displays} properties, as plain {\f1\fs20 TStrings} instance, which index will follow the {\f1\fs20 TSynTimeZone.Zone[]} internal information.
As a nice side effect, the {\f1\fs20 TSynTimeZone} binary internal storage has been found out to be very efficient, and much faster than a manual reading of the {\i Windows} registry. Complex local time calculation could be done on the server side, with no fear of breaking down your processing performances.
:184  Safe locks for multi-thread applications
:   Protect your resources
Once your application is @*multi-thread@ed, concurrent data access should be protected. Otherwise, a "@*race condition@" issue may appear: for instance, if two threads modify a variable at the same time (e.g. decrease a counter), values may become incoherent and unsafe to use. The most known symptom is the "@*deadlock@", by which the whole application appears to be blocked and unresponsive. On a server system, which is expected to run 24/7 with no maintenance, such an issue is to be avoided.
In Delphi, protection of a resource (which may be an object, or any variable) is usually done via {\i @**Critical Section@s}. A {\i critical section} is an object used to make sure, that some part of the code is executed only by one thread at a time. A {\i critical section} needs to be created/initialized before it can be used and be released when it is not needed anymore. Then, some code is protected using {\i Enter/Leave} methods, which will {\i lock} its execution: in practice, only a single thread will own the {\i critical section}, so only a single thread will be able to execute this code section, and other threads will wait until the lock is released. For best performance, the protected sections should be as small as possible - otherwise the benefit of using threads may be voided, since any other thread will wait for the thread owning the {\i critical section} to release the lock.
:   Fixing TRTLCriticalSection
In practice, you may use a {\f1\fs20 TCriticalSection} class, or the lower-level {\f1\fs20 TRTLCriticalSection} record, which is perhaps to be preferred, since it will use less memory, and could easily be included as a (protected) field to any {\f1\fs20 class} definition.
Let's say we want to protect any access to the variables a and b. Here's how to do it with the critical sections approach:
!var CS: TRTLCriticalSection;
!    a, b: integer;
!// set before the threads start
!InitializeCriticalSection(CS);
!// in each TThread.Execute:
!EnterCriticalSection(CS);
!try // protect the lock via a try ... finally block
!  // from now on, you can safely make changes to the variables
!  inc(a);
!  inc(b);
!finally
!  // end of safe block
!  LeaveCriticalSection(CS);
!end;
!// when the threads stop
!DeleteCriticalSection(CS);
In newest versions of Delphi, you may use a {\f1\fs20 @*TMonitor@} class, which will let the lock be owned by any Delphi {\f1\fs20 TObject}. Before XE5, there was some performance issue, and even now, this Java-inspired feature may not be the best approach, since it is tied to a single object, and is not compatible with older versions of Delphi (or FPC).
Eric Grange reported some years ago - see @https://www.delphitools.info/2011/11/30/fixing-tcriticalsection - that {\f1\fs20 TRTLCriticalSection} (along with {\f1\fs20 TMonitor}) suffers from a severe design flaw in which entering/leaving different {\i critical sections} can end up serializing your threads, and the whole can even end up performing worse than if your threads had been serialized. This is because it's a small, dynamically allocated object, so several {\f1\fs20 TRTLCriticalSection} memory can end up in the same CPU cache line, and when that happens, you'll have cache conflicts aplenty between the cores running the threads.
The fix proposed by Eric is dead simple:
!type
!   TFixedCriticalSection = class(TCriticalSection)
!   private
!     FDummy: array [0..95] of Byte;
!   end;
:   Introducing TSynLocker
Since we wanted to use a {\f1\fs20 TRTLCriticalSection} record instead of a {\f1\fs20 TCriticalSection} class instance, we defined a {\f1\fs20 @**TSynLocker@} record in {\f1\fs20 SynCommons.pas}:
!  TSynLocker = record
!  private
!    fSection: TRTLCriticalSection;
!  public
!    Padding: array[0..6] of TVarData;
!    procedure Init;
!    procedure Done;
!    procedure Lock;
!    procedure UnLock;
!  end;
As you can see, the {\f1\fs20 Padding[]} array will ensure that the CPU cache-line issue won't affect our object.
{\f1\fs20 TSynLocker} use is close to {\f1\fs20 TRTLCriticalSection}, with some method-oriented behavior:
!var safe: TSynLocker;
!    a, b: integer;
!// set before the threads start
!safe.Init;
!// in each TThread.Execute:
!safe.Lock
!try // protect the lock via a try ... finally block
!  // from now on, you can safely make changes to the variables
!  inc(a);
!  inc(b);
!finally
!  // end of safe block
!  safe.Unlock;
!end;
!// when the threads stop
!safe.Done;
If your purpose is to protect a method execution, you may use the {\f1\fs20 TSynLocker.ProtectMethod} function or explicit {\f1\fs20 Lock/Unlock}, as such:
!type
!  TMyClass = class
!  protected
!    fSafe: TSynLocker;
!    fField: integer;
!  public
!    constructor Create;
!    destructor Destroy; override;
!    procedure UseLockUnlock;
!    procedure UseProtectMethod;
!  end;
!
!{ TMyClass }
!
!constructor TMyClass.Create;
!begin
!!  fSafe.Init; // we need to initialize the lock
!end;
!
!destructor TMyClass.Destroy;
!begin
!!  fSafe.Done; // finalize the lock
!  inherited;
!end;
!
!procedure TMyClass.UseLockUnlock;
!begin
!!  fSafe.Lock;
!  try
!    // now we can safely access any protected field from multiple threads
!    inc(fField);
!  finally
!!    fSafe.UnLock;
!  end;
!end;
!
!procedure TMyClass.UseProtectMethod;
!begin
!!  fSafe.ProtectMethod; // calls fSafe.Lock and return IUnknown local instance
!  // now we can safely access any protected field from multiple threads
!  inc(fField);
!  // here fSafe.UnLock will be called when IUnknown is released
!end;
:   Inheriting from T*Locked
For your own classes definition, you may inherit from some classes providing a {\f1\fs20 TSynLocker} instance, as defined in {\f1\fs20 SynCommons.pas}:
!  TSynPersistentLocked = class(TSynPersistent)
!  ...
!    property Safe: TSynLocker read fSafe;
!  end;
!  TInterfacedObjectLocked = class(TInterfacedObjectWithCustomCreate)
!  ...
!    property Safe: TSynLocker read fSafe;
!  end;
!  TObjectListLocked = class(TObjectList)
!  ...
!    property Safe: TSynLocker read fSafe;
!  end;
!  TRawUTF8ListHashedLocked = class(TRawUTF8ListHashed)
!  ...
!    property Safe: TSynLocker read fSafe;
!  end;
All those classes will initialize and finalize their owned {\f1\fs20 Safe} instance, in their {\f1\fs20 constructor/destructor}.
So, we may have written our class as such:
!type
!  TMyClass = class(TSynPersistentLocked)
!  protected
!    fField: integer;
!  public
!    procedure UseLockUnlock;
!    procedure UseProtectMethod;
!  end;
!
!{ TMyClass }
!
!procedure TMyClass.UseLockUnlock;
!begin
!  fSafe.Lock;
!  try
!    // now we can safely access any protected field from multiple threads
!    inc(fField);
!  finally
!    fSafe.UnLock;
!  end;
!end;
!
!procedure TMyClass.UseProtectMethod;
!begin
!  fSafe.ProtectMethod; // calls fSafe.Lock and return IUnknown local instance
!  // now we can safely access any protected field from multiple threads
!  inc(fField);
!  // here fSafe.UnLock will be called when IUnknown is released
!end;
As you can see, the {\f1\fs20 Safe: TSynLocker} instance will be defined and handled at {\f1\fs20 TSynPersistentLocked} parent level.
:   Injecting TAutoLocker instances
Inheriting from a {\f1\fs20 TSynPersistentLocked} class (or one of its sibbling) only gives you access to a single {\f1\fs20 TSynLocker} per instance. If your class inherits from {\f1\fs20 TSynAutoCreateFields}, you may create one or several {\f1\fs20 TAutoLocker published} properties, which will be auto-created with the instance:
!type
!  TMyClass = class(TSynAutoCreateFields)
!  protected
!    fLock: TAutoLocker;
!    fField: integer;
!  public
!    function FieldValue: integer;
!  published
!    property Lock: TAutoLocker read fLock;
!  end;
!
!{ TMyClass }
!
!function TMyClass.FieldValue: integer;
!begin
!  fLock.ProtectMethod;
!  result := fField;
!  inc(fField);
!end;
!
!var c: TMyClass;
!begin
!  c := TMyClass.Create;
!  Assert(c.FieldValue=0);
!  Assert(c.FieldValue=1);
!  c.Free;
!end.
In practice, {\f1\fs20 TSynAutoCreateFields} is a very powerful way of defining @*Value objects@, i.e. objects containing nested objects or even arrays of objects. You may use its ability to create the needed {\f1\fs20 TAutoLocker} instances in an automated way. But be aware that if you serialize such an instance into JSON, its nested {\f1\fs20 TAutoLocker} properties will be serialized as void properties - which may not be the expected result.
:   Injecting IAutoLocker instances
If your class inherits from {\f1\fs20 @*TInjectableObject@}, you may define the following:
!type
!  TMyClass = class(TInjectableObject)
!  private
!    fLock: IAutoLocker;
!    fField: integer;
!  public
!    function FieldValue: integer;
!  published
!    property Lock: IAutoLocker read fLock write fLock;
!  end;
!
!{ TMyClass }
!
!function TMyClass.FieldValue: integer;
!begin
!  Lock.ProtectMethod;
!  result := fField;
!  inc(fField);
!end;
!
!var c: TMyClass;
!begin
!  c := TMyClass.CreateInjected([],[],[]);
!  Assert(c.FieldValue=0);
!  Assert(c.FieldValue=1);
!  c.Free;
!end;
Here we use dependency resolution - see @161@ - to let the {\f1\fs20 TMyClass.CreateInjected} constructor scan its {\f1\fs20 published} properties, and therefore search for a provider of {\f1\fs20 IAutoLocker}. Since {\f1\fs20 IAutoLocker} is globally registered to be resolved with {\f1\fs20 TAutoLocker}, our class will  initialize its {\f1\fs20 fLock} field with a new instance. Now we could use {\f1\fs20 Lock.ProtectMethod} to use the associated {\f1\fs20 TAutoLocker}'s {\f1\fs20 TSynLocker} critical section, as usual.
Of course, this may sounds more complicated than manual {\f1\fs20 TSynLocker} handling, but if you are writing an interface-based service - see @63@, your class may already inherit from {\f1\fs20 TInjectableObject} for its own dependency resolution, so this trick may be very convenient.
:   Safe locked storage in TSynLocker
When we fixed the potential CPU cache-line issue, do you remember that we added a padding binary buffer to the {\f1\fs20 TSynLocker} definition? Since we do not want to waste resource, {\f1\fs20 TSynLocker} gives easy access to its internal data, and allow to directly handle those values. Since it is stored as 7 slots of {\f1\fs20 variant} values, you could store any kind of data, including complex {\f1\fs20 @*TDocVariant@} document or array.
Our class may use this feature, and store its integer field value in the internal slot 0:
!type
!  TMyClass = class(TSynPersistentLocked)
!  public
!    procedure UseInternalIncrement;
!    function FieldValue: integer;
!  end;
!
!{ TMyClass }
!
!function TMyClass.FieldValue: integer;
!begin // value read will also be protected by the mutex
!  result := fSafe.LockedInt64[0];
!end;
!
!procedure TMyClass.UseInternalIncrement;
!begin // this dedicated method will ensure an atomic increase
!  fSafe.LockedInt64Increment(0,1);
!end;
Please note that we used the {\f1\fs20 TSynLocker.LockedInt64Increment()} method, since the following will not be safe:
!procedure TMyClass.UseInternalIncrement;
!begin
!  fSafe.LockedInt64[0] := fSafe.LockedInt64[0]+1;
!end;
In the above line, two locks are acquired (one per {\f1\fs20 LockedInt64} property call), so another thread may modify the value in-between, and the increment may not be as accurate as expected.
{\f1\fs20 TSynLocker} offers some dedicated properties and methods to handle this safe storage. Those expect an {\f1\fs20 Index} value, from {\f1\fs20 0..6} range:
!    property Locked[Index: integer]: Variant read GetVariant write SetVariant;
!    property LockedInt64[Index: integer]: Int64 read GetInt64 write SetInt64;
!    property LockedPointer[Index: integer]: Pointer read GetPointer write SetPointer;
!    property LockedUTF8[Index: integer]: RawUTF8 read GetUTF8 write SetUTF8;
!    function LockedInt64Increment(Index: integer; const Increment: Int64): Int64;
!    function LockedExchange(Index: integer; const Value: variant): variant;
!    function LockedPointerExchange(Index: integer; Value: pointer): pointer;
You may store a {\f1\fs20 pointer} or a reference to a {\f1\fs20 TObject} instance, if necessary.
Having such a tool-set of thread-safe methods does make sense, in the context of our framework, which offers multi-thread server abilities - see @25@.
:   Thread-safe TSynDictionary
Remember that the @200@ class is thread-safe. In fact, the {\f1\fs20 @*TSynDictionary@} methods are protected by a {\f1\fs20 TSynLocker} instance, and internal Count or TimeOuts values are actually stored within its 7 locked storage slots.
You may consider defining {\f1\fs20 TSynDictionary} instances in your business logic, or in the public API layer of your services, with proper thread safety - see @201@.
\page
:3Object-Relational Mapping
%cartoon02.png
The @**ORM@ part of the framework - see @13@ - is mainly implemented in the @!Lib\mORMot.pas@ unit. Then it will use other units (like @!Lib\mORMotSQLite3.pas@, @!Lib\mORMotDB.pas@, @!Lib\SynSQLite3.pas@ or @!Lib\SynDB.pas@) to access to the various database back-ends.
Generic access to the data is implemented by defining high-level objects as {\i Delphi} classes, descendant from a main {\f1\fs20 @**TSQLRecord@} class.
In our @*Client-Server@ ORM, those {\f1\fs20 TSQLRecord} classes can be used for at least three main purposes:
- To store and retrieve data from any database engine - for most common usage, you can forget about writing @*SQL@ queries: @*CRUD@ data access statements ({\f1\fs20 SELECT / INSERT / UPDATE /DELETE}) are all created on the fly by the {\i Object-relational mapping} (ORM) core of {\i mORMot} - see @27@ - a @*NoSQL@ engine like {\i @*MongoDB@} can even be accessed the same way - see @84@;
- To have business logic objects accessible for both the Client and Server side, in a @*REST@ful approach - see @114@;
- To fill a grid content with the proper field type (e.g. grid column names are retrieved from property names after translation, enumerations are displayed as plain text, or {\f1\fs20 boolean} as a checkbox); to create menus and reports directly from the field definition; to have edition window generated in an automated way - see @31@.
Our ORM engine has genuine advanced features like {\i @*convention over configuration@}, integrated security, local or remote access, REST JSON publishing (for AJAX or mobile clients), direct access to the database (by-passing slow {\f1\fs20 DB.pas} unit), content in-memory cache, optional audit-trail (change tracking), and integration with other parts of the framework (like @*SOA@, logging, @*authentication@...).
\page
:26 TSQLRecord fields definition
All the framework @*ORM@ process relies on the {\f1\fs20 TSQLRecord} class. This abstract {\f1\fs20 TSQLRecord} class features a lot of built-in methods, convenient to do most of the ORM process in a generic way, at record level.
It first defines a {\i @**primary key@} field, defined as {\f1\fs20 ID: @**TID@}, i.e. as {\f1\fs20 Int64} in {\f1\fs20 mORMot.pas}:
!type
!  TID = type Int64;
!  ...
!  TSQLRecord = class(TObject)
!  ...
!    property ID: TID read GetID write fID;
!  ...
In fact, our ORM relies on a {\f1\fs20 Int64} primary key, matching the {\i SQLite3} {\f1\fs20 ID}/{\f1\fs20 RowID} primary key.
You may be disappointed by this limitation, which is needed by the {\i SQLite3}'s implementation of @*Virtual Table@s - see @20@. We won't debate about a composite primary key (i.e. several fields), which is not a good idea for an ORM. In your previous RDBMS data modeling, you may be used to define a TEXT primary key, or even a GUID primary key: those kinds of keys are somewhat less efficient than an INTEGER, especially for ORM internals, since they are not monotonic. You can always define a secondary key, as {\f1\fs20 string} or {\f1\fs20 TGUID} field, if needed - using {\f1\fs20 stored @*AS_UNIQUE@} attribute as explained below.
All @*published properties@ of the {\f1\fs20 TSQLRecord} descendant classes are then accessed via @*RTTI@ in a Client-Server @*REST@ful architecture.
For example, a database {\f1\fs20 Baby} Table is defined in {\i Delphi} code as:
!/// some enumeration
!// - will be written as 'Female' or 'Male' in our UI Grid
!// - will be stored as its ordinal value, i.e. 0 for sFemale, 1 for sMale
!// - as you can see, ladies come first, here
!TSex = (sFemale, sMale);
!
!/// table used for the Babies queries
!TSQLBaby = class(TSQLRecord)
!  private
!    fName: RawUTF8;
!    fAddress: RawUTF8;
!    fBirthDate: TDateTime;
!    fSex: TSex;
!  published
!    property Name: RawUTF8 read fName write fName;
!    property Address: RawUTF8 read fAddress write fAddress;
!    property BirthDate: TDateTime read fBirthDate write fBirthDate;
!    property Sex: TSex read fSex write fSex;
!end;
By adding this {\f1\fs20 TSQLBaby} class to a {\f1\fs20 @*TSQLModel@} instance, common for both Client and Server, the corresponding {\i Baby} table is created by the Framework in the database engine ({\i @*SQLite3@} natively or any external database). All @*SQL@ work ('{\f1\fs20 CREATE TABLE ...}') is done by the framework. Just code in Pascal, and all is done for you. Even the needed indexes will be created by the ORM. And you won't miss any ' or ; in your SQL query any more.
\page
The following {\f1\fs20 @**published properties@} types are handled by the @*ORM@, and will be converted as specified to database content (in {\i SQLite3}, an INTEGER is an {\f1\fs20 Int64}, FLOAT is a double, TEXT is an @*UTF-8@ encoded text):
|%24%14%64
|\b Delphi|SQLite3|Remarks\b0
|{\f1\fs20 byte}|INTEGER|
|{\f1\fs20 word}|INTEGER|
|{\f1\fs20 integer}|INTEGER|
|{\f1\fs20 cardinal}|INTEGER|
|{\f1\fs20 Int64}|INTEGER|
|{\f1\fs20 boolean}|INTEGER|0 is {\f1\fs20 false}, anything else is {\f1\fs20 true}
|enumeration|INTEGER|store the ordinal value of the @*enumerated@ item(i.e. starting at 0 for the first element)
|set|INTEGER|each bit corresponding to an enumerated item (therefore a set of up to 64 elements can be stored in such a field)
|{\f1\fs20 single}|FLOAT|
|{\f1\fs20 @*double@}|FLOAT|
|{\f1\fs20 extended}|FLOAT|stored as {\f1\fs20 double} (precision lost)
|{\f1\fs20 @*currency@}|FLOAT|safely converted to/from {\f1\fs20 currency} type with fixed decimals, without rounding error
|{\f1\fs20 @*RawUTF8@}|TEXT|this is the {\b preferred} field type for storing some textual content in the ORM
|{\f1\fs20 WinAnsiString}|TEXT|{\i WinAnsi} char-set (code page 1252) in {\i Delphi}
|{\f1\fs20 RawUnicode}|TEXT|{\i UCS2} char-set in {\i Delphi}, as {\f1\fs20 AnsiString}
|{\f1\fs20 @*WideString@}|TEXT|{\i UCS2} char-set, as COM BSTR type (Unicode in all version of {\i Delphi})
|{\f1\fs20 @*SynUnicode@}|TEXT|Will be either {\f1\fs20 WideString} before {\i Delphi} 2009, or {\f1\fs20 UnicodeString} later
|{\f1\fs20 string}|TEXT|Not to be used before {\i Delphi} 2009 (unless you may loose some data during conversion) - {\f1\fs20 RawUTF8} is preferred in all cases
|{\f1\fs20 @*TDateTime@}|TEXT|@*ISO 8601@ encoded date time, with second resolution
|{\f1\fs20 @*TDateTimeMS@}|TEXT|@*ISO 8601@ encoded date time, with millisecond resolution
|{\f1\fs20 @*TTimeLog@}|INTEGER|as proprietary fast {\f1\fs20 Int64} date time
|{\f1\fs20 @*TModTime@}|INTEGER|the server date time will be stored when a record is modified (as proprietary fast {\f1\fs20 Int64})
|{\f1\fs20 @*TCreateTime@}|INTEGER|the server date time will be stored when a record is created (as proprietary fast {\f1\fs20 Int64})
|{\f1\fs20 @*TUnixTime@}|INTEGER|timestamp stored as second-based Unix Time (i.e. the 64-bit number of seconds since 1970-01-01 00:00:00 UTC)
|{\f1\fs20 @*TUnixMSTime@}|INTEGER|timestamp stored as millisecond-based Unix Time (i.e. the 64-bit number of milliseconds since 1970-01-01 00:00:00 UTC)
|{\f1\fs20 @*TSQLRecord@}|INTEGER|32-bit {\f1\fs20 RowID} pointing to another record (warning: the field value contains {\f1\fs20 pointer(RowID)}, not a valid object instance - the record content must be retrieved with late-binding via its {\f1\fs20 ID} using a {\f1\fs20 PtrInt(Field)} typecast or the {\f1\fs20 Field.ID} method), or by using e.g. {\f1\fs20 @*CreateJoined@()} - 64-bit under {\i Win64}
|{\f1\fs20 @*TID@}|INTEGER|64-bit {\f1\fs20 RowID} pointing to another record, but without any information about the corresponding table
|{\f1\fs20 @*TSQLRecordMany@}|nothing|data is stored in a separate {\i pivot} table; this is a particular case of {\f1\fs20 TSQLRecord}: it won't contain {\f1\fs20 pointer(RowID)}, but an instance)
|{\f1\fs20 @*TRecordReference@}\line {\f1\fs20 @*TRecordReferenceToBeDeleted@}|INTEGER|able to join any row on any table of the model, by storing both {\f1\fs20 ID} and {\f1\fs20 TSQLRecord} class type in a {\f1\fs20 @*RecordRef@}-like {\f1\fs20 Int64} value, with automatic reset to 0 (for {\f1\fs20 TRecordReference}) or row deletion (for {\f1\fs20 TRecordReferenceToBeDeleted}) when the pointed record is deleted
|{\f1\fs20 @*TSessionUserID@}|INTEGER|64-bit {\f1\fs20 RowID} of the {\f1\fs20 TSQLAuthUser} currently logged with the active session
|{\f1\fs20 @*TPersistent@}|TEXT|@*JSON@ object ({\f1\fs20 ObjectToJSON})
|{\f1\fs20 @*TCollection@}|TEXT|JSON array of objects ({\f1\fs20 ObjectToJSON})
|{\f1\fs20 @*TObjectList@}|TEXT|JSON array of objects ({\f1\fs20 ObjectToJSON}) - see {\f1\fs20 TJSONSerializer. @*RegisterClassForJSON@} @71@
|{\f1\fs20 @*TStrings@}|TEXT|JSON array of string ({\f1\fs20 ObjectToJSON})
|{\f1\fs20 @*TRawUTF8List@}|TEXT|JSON array of string ({\f1\fs20 ObjectToJSON})
|any {\f1\fs20 @*TObject@}|TEXT|See {\f1\fs20 TJSONSerializer. @*RegisterCustomSerializer@} @52@
|{\f1\fs20 @*TSQLRawBlob@}|@*BLOB@|This type is an alias to {\f1\fs20 @*RawByteString@}
|{\i @*dynamic array@s}|BLOB|in the {\f1\fs20 TDynArray.SaveTo} binary format
|{\f1\fs20 variant}|TEXT|numerical or text in JSON, or @80@ for JSON objects or arrays
|{\f1\fs20 @*TNullableInteger@}|INTEGER|{\i Nullable} {\f1\fs20 Int64} value - see @177@
|{\f1\fs20 @*TNullableBoolean@}|INTEGER|{\i Nullable} {\f1\fs20 boolean} (0/1/NULL) value - see @177@
|{\f1\fs20 @*TNullableFloat@}|FLOAT|{\i Nullable} {\f1\fs20 double} value - see @177@
|{\f1\fs20 @*TNullableCurrency@}|FLOAT|{\i Nullable} {\f1\fs20 currency} value - see @177@
|{\f1\fs20 @*TNullableDateTime@}|TEXT|{\i Nullable} @*ISO 8601@ encoded date time - see @177@
|{\f1\fs20 @*TNullableTimeLog@}|INTEGER|{\i Nullable} @*TTimeLog@ value - see @177@
|{\f1\fs20 @*TNullableUTF8Text@}|TEXT|{\i Nullable} Unicode text value - see @177@
|{\f1\fs20 record}|TEXT|JSON string or object, directly handled since {\i Delphi} XE5, or as defined in code by overriding {\f1\fs20 TSQLRecord.} {\f1\fs20 InternalRegisterCustomProperties} for prior versions
|{\f1\fs20 @*TRecordVersion@}|INTEGER|64-bit revision number, which will be monotonically updated each time the object is modified, to allow remote synchronization - see @147@
|%
\page
:178  Property Attributes
Some additional attributes may be added to the {\f1\fs20 published} field definitions:
- If the property is marked as {\f1\fs20 stored @**AS_UNIQUE@} (i.e. {\f1\fs20 stored false}), it will be created as UNIQUE in the database (i.e. a SQL index will be created and uniqueness of the value will be checked at insert/update);
- For a dynamic array field, the {\f1\fs20 index} number can be used for the {\f1\fs20 TSQLRecord. DynArray(DynArrayFieldIndex)} method to create a {\f1\fs20 TDynArray} wrapper mapping the dynamic array data;
- For a {\f1\fs20 RawUTF8 / string / WideString / WinAnsiString} field of an "external" class - i.e. a TEXT field stored in a remote {\f1\fs20 @*SynDB@.pas}-based database - see @145@, the {\f1\fs20 @*index@} number will be used to define the maximum character size of this field, when creating the corresponding column in the database (@*SQLite3@ or {\i @*PostgreSQL@} does not have any such size expectations).
For instance, the following {\f1\fs20 class} definition will create an index for its {\f1\fs20 SerialNumber} property (up to 30 characters long if stored in an external database), and will expect a link to a model of diaper ({\f1\fs20 TSQLDiaperModel}) and the baby which used it ({\f1\fs20 TSQLBaby}). An {\f1\fs20 ID} / {\f1\fs20 RowID} column will be always available (from {\f1\fs20 TSQLRecord}), so in this case, you will be able to make a fast lookup for a particular diaper from either its internal {\i mORmot} ID, or its official unique serial number:
!/// table used for the Diaper queries
!TSQLDiaper = class(TSQLRecord)
!  private
!    fSerialNumber: RawUTF8;
!    fModel: TSQLDiaperModel;
!    fBaby: TSQLBaby;
!  published
!    property SerialNumber: RawUTF8
!!     index 30
!      read fSerialNumber write fSerialNumber
!!     stored AS_UNIQUE;
!    property Model: TSQLDiaperModel read fModel write fModel;
!    property Baby: TSQLBaby read fBaby write fBaby;
!end;
Note that {\f1\fs20 @*TTNullableUTF8Text@} kind of property will follow the same {\f1\fs20 index ###} attribute interpretation.
:  Text fields
In practice, the generic {\f1\fs20 string} type is handled (as {\f1\fs20 UnicodeString} under {\i Delphi} 2009 and later), but you may loose some content if you're working with pre-Unicode version of {\i Delphi} (in which {\f1\fs20 string = AnsiString} with the current system code page). So we won't recommend its usage.
The natural {\i Delphi} type to be used for TEXT storage in our framework is {\f1\fs20 @*RawUTF8@} as introduced for @32@. All business process should better use {\f1\fs20 RawUTF8} variables and methods (you have all necessary functions in {\f1\fs20 SynCommons.pas}), then you should explicitly convert the {\f1\fs20 RawUTF8} content into a string using {\f1\fs20 U2S / S2U} from {\f1\fs20 mORMoti18n.pas} or {\f1\fs20 StringToUTF8 / UTF8ToString} which will handle proper char-set conversion according to the current @*i18n@ settings. On Unicode version of {\i Delphi} (starting with {\i Delphi} 2009), you can directly assign a {\f1\fs20 string / UnicodeString} value to / from a {\f1\fs20 RawUTF8}, but this implicit conversion will be slightly slower than our {\f1\fs20 StringToUTF8 / UTF8ToString} functions. With pre-Unicode version of {\i Delphi} (up to {\i Delphi} 2007), such direct assignation will probably loose data for all non ASCII 7 bit characters, so an explicit call to {\f1\fs20 StringToUTF8 / UTF8ToString} functions is required.
You will find in {\f1\fs20 SynCommons.pas} unit all low-level {\f1\fs20 RawUTF8} processing functions and classes, to be used instead of any {\f1\fs20 SysUtils.pas} functions. The {\f1\fs20 mORMot} core implementation about {\f1\fs20 RawUTF8} is very optimized for speed and multi-threading, so it is recommended not to use {\f1\fs20 string} in your code, unless you access to the VCL / User Interface layer.
Having such a dedicated {\f1\fs20 RawUTF8} type will also ensure that you are not leaking your domain from its business layer to the presentation layer, as defined with @7@:
\graph ArchiStringTier Strings in Domain Driven Design n-Tier Architecture
\Presentation Tier\Application Tier
\Application Tier\Business Logic Tier
\Business Logic Tier\Data Tier
\string\RawUTF8
\RawUTF8\ RawUTF8
\ RawUTF8\  RawUTF8
\Presentation Tier=string
\Application Tier=RawUTF8
\Business Logic Tier= RawUTF8
\Data Tier=  RawUTF8
\
For additional information about @*UTF-8@ handling in the framework, see @32@.
:174  Date and time fields
{\i Delphi} {\f1\fs20 @*TDateTime@} and {\f1\fs20 @*TDateTimeMS@} properties will be stored as @*ISO 8601@ text in the database, with seconds and milliseconds resolution. See @175@ for details about this text encoding.
As alternatives, {\f1\fs20 @*TTimeLog@ / @**TModTime@ / @**TCreateTime@} offer a proprietary fast {\f1\fs20 Int64} date time format, which will map the {\f1\fs20 @*TTimeLogBits@} record type, as defined in {\f1\fs20 SynCommons.pas} unit.
This format will be very fast for comparing dates or convert into/from text, and will be stored as INTEGER in the database, therefore more efficiently than plain ISO 8601 text as for {\f1\fs20 TDateTime} fields.
In practice, {\f1\fs20 TModTime} and {\f1\fs20 TCreateTime} values are inter-exchangeable with {\f1\fs20 TTimeLog}. They are just handled with a special care by the ORM, so that their associated field value will be updated with the current UTC timestamp, for every {\f1\fs20 TSQLRecord} modification (for {\f1\fs20 TModTime}), or at entry creation (for {\f1\fs20 TCreateTime}). The time value stored is in fact the UTC timestamp, as returned from the current REST Server: in fact, when any REST client perform a connection, it will retrieve any time offset from the REST Server, which will be used to store a consistent time value across all Clients.
You may also define a {\f1\fs20 @*TUnixTime@} property, which will store the number of seconds since 1970-01-01 00:00:00 UTC as INTEGER in the database, and serialized as 64-bit JSON number - or {\f1\fs20 @*TUnixMSTime@} if you expect milliseconds resolution. This encoding has the benefit of being handled by {\i SQlite3} date/time functions, and interoperable with most third-party languages.
:  TSessionUserID field
If you define a {\f1\fs20 @**TSessionUserID@} published property, this field will be automatically filled at creation or modification of the {\f1\fs20 TSQLRecord} with the current {\f1\fs20 TSQLAuthUser.ID} value of the active session. If no session has been initialized from the client side, {\f1\fs20 0} will be stored.
By design, and similar to {\f1\fs20 @*TModTime@} fields, you should use the @*ORM@ PUT/POST @*CRUD@ methods to compute this field value: manual SQL statements (like {\f1\fs20 UPDATE Table SET Column=0}) won't set its content. Also, it is up to the client to fill the {\f1\fs20 TSessionUserID} fields before sending their content to the server - the Delphi and cross-platform ORM clients will perform this assignment.
:  Enumeration fields
{\i Enumerations} should be mapped as INTEGER, i.e. via {\f1\fs20 ord(aEnumValue)} or {\f1\fs20 TEnum(aIntegerValue)}.
{\i Enumeration sets} should be mapped as INTEGER, with {\f1\fs20 byte/word/integer} type, according to the number of elements in the set: for instance, {\f1\fs20 byte(aSetValue)} for up to 8 elements, {\f1\fs20 word(aSetValue)} for up to 16 elements, and {\f1\fs20 integer(aSetValue)} for up to 32 elements in the set.
:  Floating point and Currency fields
For standard floating-point values, the framework natively handles the {\f1\fs20 @*double@} and {\f1\fs20 @**currency@} kind of variables.
In fact, {\f1\fs20 double} is the native type handled by most database providers - it is also native to the SSE set of opcodes of newer CPUs (as handled by {\i Delphi} XE 2 in @*64-bit@ mode). Lack of {\f1\fs20 extended} should not be problematic (if it is mandatory, a dedicated set of mathematical classes should be preferred to a database), and could be implemented with the expected precision via a TEXT field (or a BLOB mapped by a @*dynamic array@).
The {\f1\fs20 currency} type is the standard {\i Delphi} type to be used when storing and handling monetary values, native to the x87 FPU - when it comes to money, a dedicated type is worth the cost in a "rich man's world". It will avoid any rounding problems, assuming exact 4 decimals precision. It is able to safely store numbers in the range -922337203685477.5808 .. 922337203685477.5807. Should be enough for your pocket change.
As stated by the official {\i Delphi} documentation:
{\i {\f1\fs20 Currency} is a fixed-point data type that minimizes rounding errors in monetary calculations. On the Win32 platform, it is stored as a scaled 64-bit integer with the four least significant digits implicitly representing decimal places. When mixed with other real types in assignments and expressions, {\f1\fs20 Currency} values are automatically divided or multiplied by 10000.}
In fact, this type matches the corresponding {\f1\fs20 OLE} and {\f1\fs20 .Net} implementation of {\f1\fs20 currency}. It is still implemented the same in the {\i Win64} platform (since XE 2). The {\f1\fs20 Int64} binary representation of the {\f1\fs20 currency} type (i.e. {\f1\fs20 value*10000} as accessible via a typecast like {\f1\fs20 PInt64(@aCurrencyValue)^}) is a safe and fast implementation pattern.
In our framework, we tried to avoid any unnecessary conversion to float values when dealing with {\f1\fs20 currency} values. Some dedicated functions have been implemented - see @33@ - for fast and secure access to {\f1\fs20 currency} published properties via @*RTTI@, especially when converting values to or from @*JSON@ text. Using the {\f1\fs20 Int64} binary representation can be not only faster, but also safer: you will avoid any rounding problem which may be introduced by the conversion to a float type. For all database process, especially with external engines, the {\f1\fs20 SynDB.pas} units will try to avoid any conversion to/from {\f1\fs20 @*double@} for the dedicated {\f1\fs20 ftCurrency} columns.\line Rounding issues are a nightmare to track in production - it sounds safe to have a framework handling natively a {\f1\fs20 currency} type from the ground up.
:  TSQLRecord fields
It is worth saying that {\f1\fs20 @*TSQLRecord@} published properties are not by default {\f1\fs20 class} instances, as with regular {\i Delphi} code. After running {\f1\fs20 TSQLRecord.Create()} or {\f1\fs20 CreateAndFillPrepare()} constructors, you should never call {\f1\fs20 aMyRecord.AnotherRecord.Property} directly, or you will raise an {\i Access Violation}.
In fact, {\f1\fs20 TSQLRecord} published properties definition is used to define "@*one to many@" or "@*one to one@" relationships between tables. As a consequence, the nested {\f1\fs20 AnotherRecord} property won't be a true {\f1\fs20 class} instance, but one ID trans-typed as {\f1\fs20 TSQLRecord}.
Only exception to this rule is {\f1\fs20 TSQLRecordMany} kind of published properties, which, by design, are true instances, needed to access the pivot table data of "@*many to many@" relationship. The ORM will {\i auto-instantiate} all {\f1\fs20 TSQLRecordMany} published properties, then release them at {\f1\fs20 Destroy} - so you do not need to maintain their life time.
Note that you may use e.g. {\f1\fs20 TSQLRecord.@*CreateJoined@()} constructor to {\i auto-instantiate} and load all {\f1\fs20 TSQLRecord} published properties at once, then release them at {\f1\fs20 Destroy}. - see @129@.
The ORM will automatically perform the following optimizations for {\f1\fs20 TSQLRecord} published fields:
- An {\i index} will be created on the database, for the corresponding column;
- When a referenced record is deleted, the ORM will detect it and automatically set all published properties pointing to this record to 0.
In fact, the ORM won't define a {\f1\fs20 ON DELETE SET DEFAULT} foreign key via SQL: this feature won't be implemented at RDBMS level, but emulated {\i at ORM level}.
See @70@ for more details about how to work with {\f1\fs20 @*TSQLRecord@} published properties.
:  TID fields
{\f1\fs20 @*TSQLRecord@} published properties do match a class instance pointer, so are 32-bit (at least for {\i Win32/Linux32} executables). Since the {\f1\fs20 TSQLRecord.ID} field is declared as {\f1\fs20 @*TID@ = Int64}, we may loose information if the stored {\f1\fs20 ID} is greater than 2,147,483,647 (i.e. a signed 32-bit value).
You can define a published property as {\f1\fs20 TID} to store any value of our @*primary key@, i.e. up to 9,223,372,036,854,775,808. Note that in this case, there is no information about the joined table.
As a consequence, the ORM will perform the following optimizations for {\f1\fs20 TID} fields:
- An {\i index} will be created on the database, for the corresponding column;
- When a referenced record is deleted, the ORM {\i won't do anything}, since it has no information about the table to track - this is the main difference with {\f1\fs20 TSQLRecord} published property.
You can optionally specify the associated table, using a custom {\f1\fs20 TID} type for the published property definition. In this case, you will sub-class {\f1\fs20 TID}, using {\f1\fs20 {\i tableName}ID} as naming convention.\line For instance, if you define:
! type
!   TSQLRecordClientID = type TID;
!   TSQLRecordClientToBeDeletedID = type TID;
!
!   TSQLOrder = class(TSQLRecord)
!   ...
!   property Client: TID
!     read fClient write fClient;
!   property OrderedBy: TSQLRecordClientID
!     read fOrderedBy write fOrderedBy;
!   property OrderedByCascade: TSQLRecordClientToBeDeletedID
!     read fOrderedByCascade write fOrderedByCascade;
!   ...
Those three published fields will be able to store a {\f1\fs20 Int64} foreign key, and the ORM will ensure a corresponding {\i index} is created on the database, to speedup search on their values.\line But their type - {\f1\fs20 TID}, {\f1\fs20 TSQLRecordClientID}, or {\f1\fs20 TSQLRecordClientToBeDeletedID} - will define how the deletion process will be processed.
By using the generic {\f1\fs20 TID} type, the first {\f1\fs20 Client} property won't have any reference to any table, so no deletion tracking will take place.
On the other hand, {\i following the type naming convention}, the others {\f1\fs20 OrderedBy} and {\f1\fs20 OrderedByCascade} properties will be associated with the {\f1\fs20 TSQLRecordClient} table of the data model.\line In fact, the ORM will retrieve the {\f1\fs20 'TSQLRecordClientID'} or {\f1\fs20 'TSQLRecordClientToBeDeletedID'}  type names, and search for a {\f1\fs20 TSQLRecord} associated by trimming {\f1\fs20 *[ToBeDeleted]ID}, which is {\f1\fs20 TSQLRecordClient} in this case.\line As a result, the ORM will be able to track any {\f1\fs20 TSQLRecordClient} deletion: for any row pointing to the deleted record, it will ensure that this {\f1\fs20 OrderedBy} property will be reset to 0, or that the row containing the {\f1\fs20 OrderedByCascade} property will be deleted. Note that the framework won't define a {\f1\fs20 ON DELETE SET DEFAULT} or {\f1\fs20 ON DELETE CASCADE} foreign key via SQL, but emulate them {\i at ORM level}.
:148  TRecordReference and TRecordReferenceToBeDeleted
{\f1\fs20 TSQLRecord} or {\f1\fs20 TID} published properties are associated with a single {\f1\fs20 TSQLRecord} joined table. You could use {\f1\fs20 @**TRecordReference@} or {\f1\fs20 @**TRecordReferenceToBeDeleted@} published properties to store a reference to any record on any table of the data model.
In fact, such properties will store in a {\f1\fs20 Int64} value a reference to both a {\f1\fs20 TSQLRecord} class (therefore defining a table), and one {\f1\fs20 ID} (to define the row).
You could later on use e.g. {\f1\fs20 @*TSQLRest@.Retrieve(Reference)} to get a record content in one step.
One {\b important note} is to remember that the table reference is stored as an index to the {\f1\fs20 TSQLRecord} class in the associated {\f1\fs20 TSQLModel}.\line As a consequence, for such {\f1\fs20 TRecordReference*} properties to work as expected, you should ensure:
- That the order of {\f1\fs20 TSQLRecord} classes in the {\f1\fs20 TSQLModel} {\b do not change} after any model modification: otherwise, all previously stored {\f1\fs20 TRecordReference*} values may point to a wrong record;
- That both Client and Server side {\b share the same model} - at least for the {\f1\fs20 TSQLRecord} classes which are used with {\f1\fs20 TRecordReference*}.
Depending on the type, the ORM will track the deletion of the pointed record:
- {\f1\fs20 TRecordReference} fields will be reset to 0 - emulating {\f1\fs20 ON DELETE SET DEFAULT} foreign key SQL declaration;
- {\f1\fs20 TRecordReferenceToBeDeleted} will delete the whole record - emulating {\f1\fs20 ON DELETE CASCADE} foreign key SQL declaration.
Just like with {\f1\fs20 TSQLRecord} or {\f1\fs20 {\i TSQLRecordClassName}[ToBeDeleted]ID} fields, this deletion tracking is not defined at RDBMS level, but emulated {\i at ORM level}.
In order to work easily with {\f1\fs20 TRecordReference} values (which are in fact plain {\f1\fs20 Int64} values), you could transtype them into the {\f1\fs20 @**RecordRef@()} record, and access the stored information via a set of helper methods. See @128@ for an example of use of such {\f1\fs20 TRecordReference} in a data model, e.g. the {\f1\fs20 AssociatedRecord} property of {\f1\fs20 TSQLAuditTrail}.
:  TSQLRecord, TID, TRecordReference deletion tracking
To sum up all possible foreign key reference available by the framework, check out this table:
|%35%8%9%20%28
|\b Type Definition|Index|Tables|Deletion Tracking|Emulated SQL\b0
|{\f1\fs20 TSQLRecord}|Yes|One|Field reset to 0|ON DELETE SET DEFAULT
|{\f1\fs20 TID}|Yes|No|None|None
|{\f1\fs20 T{\i ClassName}ID}|Yes|One|Field reset to 0|ON DELETE SET DEFAULT
|{\f1\fs20 T{\i ClassName}ToBeDeletedID}|Yes|One|Row deleted|ON DELETE CASCADE
|{\f1\fs20 TRecordReference}|Yes|All|Field reset to 0|ON DELETE SET DEFAULT
|{\f1\fs20 TRecordReferenceToBeDeleted}|Yes|All|Row deleted|ON DELETE CASCADE
|%
It is worth saying that this deletion tracking is not defined at RDBMS level, but {\i at ORM level}.\line As a consequence, it will work with any kind of databases, including @82@. In fact, RDBMS engines do not allow defining such {\f1\fs20 ON DELETE} trigger on several tables, whereas {\i mORMot} handles such composite references as expected for {\f1\fs20 TRecordReference}.\line Since this is not a database level tracking, but only from a {\i mORMot} server, if you still use the database directly from legacy code, ensure that you will take care of this tracking, perhaps by using a @*SOA@ service instead of direct SQL statements.
:  Variant fields
The ORM will store {\f1\fs20 variant} fields as TEXT in the database, serialized as JSON.
At loading, it will check their content:
- If some custom {\f1\fs20 variant} types are registered (e.g. {\i @*MongoDB@} custom objects), they will be recognized as such (with @*extended syntax@, if applying);
- It will create a @80@ instance if the stored TEXT is a JSON object or array;
- It will create a numerical value ({\f1\fs20 integer} or {\f1\fs20 @*double@}) if the stored text has the corresponding layout;
- Otherwise, it will create a {\f1\fs20 string} value.
Since all data is stored as TEXT in the column, your queries shall ensure that any SQL WHERE statement handles it as expected (e.g. with a conversion to number before comparison). Even if {\i SQLite3} is able to affect a column type for each row (i.e. store a {\f1\fs20 variant} as in {\i Delphi} code), we did not use this feature, since we wanted our framework to work with all databases - and {\i @*SQLite3@} is quite alone having this feature.
At JSON level, {\f1\fs20 variant} fields will be transmitted as JSON text or number, depending on the stored value.
If you use a {\i MongoDB} external @*NoSQL@ database - see @84@, such {\f1\fs20 variant} field will not be stored as JSON text, but as true BSON documents. So you will be able to apply all the advanced search and indexing abilities of this database engine, if needed.
:  Record fields
Since {\i Delphi} XE5, you can define and work directly with published record properties of {\f1\fs20 @*TSQLRecord@}:
!  TSQLMyRecord = class(TSQLRecordPeople)
!  protected
!    fGUID: TGUID;
!  published
!    property GUID: TGUID read fGUID write fGUID index 38;
!  end;
The record will be serialized as JSON - here @*TGUID@ will be serialized as a JSON string - then will be stored as TEXT column in the database.\line We specified an {\f1\fs20 @*index@ 38} attribute to state that this column will contain up to 38 characters, when stored on an external database - see @145@.
Published properties of {\i records} are handled by our code, but {\i Delphi} doesn't create the corresponding @*RTTI@ for such properties before {\i Delphi} XE5.\line So {\f1\fs20 record} published properties, as defined in the above class definition, won't work directly for older versions of {\i Delphi}, or {\i @*FreePascal@}.
You could use a {\i @*dynamic array@} with only one element, in order to handle records within your {\f1\fs20 TSQLRecord} class definition - see @21@. But it may be confusing.
If you want to work with such properties before {\i Delphi} XE5, you can override the {\f1\fs20 TSQLRecord.InternalRegisterCustomProperties()} virtual method of a given table, to explicitly define a {\f1\fs20 record} property.
For instance, to register a {\f1\fs20 @*GUID@} property mapping a {\f1\fs20 TSQLMyRecord.fGUID: TGUID} field:
! type
!   TSQLMyRecord = class(TSQLRecord)
!   protected
!     fGUID: TGUID;
!     class procedure InternalRegisterCustomProperties(Props: TSQLRecordProperties); override;
!   public
!     property GUID: TGUID read fGUID write fGUID;
!   end;
!
!{ TSQLMyRecord }
!
!class procedure TSQLMyRecord.InternalRegisterCustomProperties(
!  Props: TSQLRecordProperties);
!begin
!  Props.RegisterCustomPropertyFromTypeName(self,'TGUID','GUID',
!    @TSQLRecordCustomProps(nil).fGUID,[aIsUnique],38);
!end;
You may call {\f1\fs20 Props.RegisterCustomPropertyFromRTTI()}, supplying the {\f1\fs20 TypeInfo()} pointer, for a record containing reference-counted fields like {\f1\fs20 string}, {\f1\fs20 variant} or nested dynamic arrays. Of course, any custom JSON serialization of the given {\f1\fs20 record} type - see @51@ - will be supported.
Those custom {\f1\fs20 record} registration methods will define either:
- TEXT serialization, for {\f1\fs20 RegisterCustomPropertyFromRTTI()} or {\f1\fs20 RegisterCustomPropertyFromTypeName()};
- BLOB serialization, for {\f1\fs20 RegisterCustomRTTIRecordProperty()} or {\f1\fs20 RegisterCustomFixedSizeRecordProperty()}.
:  BLOB fields
In fact, several kind of properties will be stored as @**BLOB@ in the database backend:
- {\f1\fs20 @*TSQLRawBlob@} properties are how you store your binary data, e.g. images or documents;
- {\i @*dynamic array@s} (saved in the {\f1\fs20 TDynArray.SaveTo} binary format);
- {\f1\fs20 record} which were explicitly registered as BLOB columns.
By default, both {\i dynamic arrays} and BLOB {\i record} content will be retrieved from the database, encoded as @*Base64@ text.
But {\f1\fs20 @*TSQLRawBlob@} properties will be transmitted as @*REST@ful separate resources, as required by the REST scheme. For instance, it means that a first request will retrieve all "simple" fields as JSON, then some other requests are needed to retrieve each BLOB fields as a binary buffer. As a result, {\f1\fs20 @*TSQLRawBlob@} won't be transmitted by default, to spare transmission bandwidth and resources.
You can change this default behavior, by setting:
- Either {\f1\fs20 TSQLRestClientURI.@**ForceBlobTransfert@: boolean} property, to force the transfert of all BLOBs of all the tables of the data model - this is what is done e.g. for the {\i SynFile} main demo - see later in this document;
- Or via {\f1\fs20 TSQLRestClientURI.TSQLRestClientURI.ForceBlobTransfertTable[]} property, for a specified table of the model.
:177  TNullable* fields for NULL storage
In {\i Delphi}, nullable types do not exist, as they do for instance in C#, via the {\f1\fs20 int?} kind of definition.\line But at SQL and @*JSON@ levels, the @*NULL@ values do exist and are expected to be available from our ORM.
In {\i @*SQLite3@} itself, NULL is handled as stated in @http://www.sqlite.org/lang_expr.html (see e.g. {\f1\fs20 IS} and {\f1\fs20 IS NOT} operators).\line It is worth noting that NULL handling is not consistent among all existing database engines, e.g. when you are comparing NULL with non NULL values... so we recommend using it with care in any database statements, or only with proper (unit) testing, when you switch from one database engine to another.
By default, in the {\i mORMot} ORM/SQL code, NULL will appear only in case of a BLOB storage with a size of {\f1\fs20 0} bytes. Otherwise, you should not see it as a value, in most used types - see @26@.
Null-oriented value types have been implemented in our framework, since the object pascal language does not allow defining a nullable type (yet). We choose to store those values as {\f1\fs20 variant}, with a set of {\f1\fs20 @**TNullable@*} dedicated types, as defined in {\f1\fs20 mORMot.pas}:
!type
!  TNullableInteger = type variant;
!  TNullableBoolean = type variant;
!  TNullableFloat = type variant;
!  TNullableCurrency = type variant;
!  TNullableDateTime = type variant;
!  TNullableTimeLog = type variant;
!  TNullableUTF8Text = type variant;
In order to define a {\f1\fs20 NULLable} column of such types, you could use them as types for your {\f1\fs20 TSQLRecord} class definition:
!type
!  TSQLNullableRecord = class(TSQLRecord)
!  protected
!    fInt: TNullableInteger;
!    fBool: TNullableBoolean;
!    fFlt: TNullableFloat;
!    fCurr: TNullableCurrency;
!    fDate: TNullableDateTime;
!    fTimestamp: TNullableTimeLog;
!    fCLOB: TNullableUTF8Text;
!    fText: TNullableUTF8Text;
!  published
!    property Int: TNullableInteger read fInt write fInt;
!    property Bool: TNullableBoolean read fBool write fBool;
!    property Flt: TNullableFloat read fFlt write fFlt;
!    property Curr: TNullableCurrency read fCurr write fCurr;
!    property Date: TNullableDateTime read fDate write fDate;
!    property Timestamp: TNullableTimeLog read fTimestamp write fTimestamp;
!    property CLOB: TNullableUTF8Text read fCLOB write fCLOB;
!    property Text: TNullableUTF8Text index 32 read fText write fText;
!  end;
Such a class will let the ORM handle SQL NULL values as expected, i.e. returning a {\f1\fs20 null} variant value, or an integer/number/text value if there is something stored. Of course, the corresponding column in the database will have the expected data type, e.g. a {\f1\fs20 NULLABLE INTEGER} for {\f1\fs20 TNullableInteger} property.
Note that {\f1\fs20 TNullableUTF8Text} is defined as a {\f1\fs20 RawUTF8} usual field - see @178@. That is, without any size limitation by default (as for the {\f1\fs20 CLOB} property), or with an explicit size limitation using the {\f1\fs20 index ###} attribute (as for {\f1\fs20 Text} property, which will be converted as a {\f1\fs20 VARCHAR(32)} SQL column).
You could use the following wrapper functions to create a {\f1\fs20 TNullable*} value from any non-nullable standard Delphi value:
!function NullableInteger(const Value: Int64): TNullableInteger;
!function NullableBoolean(Value: boolean): TNullableBoolean;
!function NullableFloat(const Value: double): TNullableFloat;
!function NullableCurrency(const Value: currency): TNullableCurrency;
!function NullableDateTime(const Value: TDateTime): TNullableDateTime;
!function NullableTimeLog(const Value: TTimeLog): TNullableTimeLog;
!function NullableUTF8Text(const Value: RawUTF8): TNullableUTF8Text;
Some corresponding constants do match the expected {\f1\fs20 null} value for each kind, with strong typing (to be used for @*FPC@ compatibility, which does not allow direct assignment of a plain {\f1\fs20 null: variant} to a {\f1\fs20 TNullable* = type variant} property):
!var
!  NullableIntegerNull: TNullableInteger absolute NullVarData;
!  NullableBooleanNull: TNullableBoolean absolute NullVarData;
!...
You could check for a {\f1\fs20 TNullable*} value to contain null, using the following functions:
!function NullableIntegerIsEmptyOrNull(const V: TNullableInteger): Boolean;
!function NullableBooleanIsEmptyOrNull(const V: TNullableBoolean): Boolean;
!...
Or retrieve a Delphi non-nullable value in one step, using the corresponding wrappers:
!function NullableIntegerToValue(const V: TNullableInteger; out Value: Int64): Boolean;
!function NullableBooleanToValue(const V: TNullableBoolean; out Value: Boolean): Boolean;
!...
!function NullableIntegerToValue(const V: TNullableInteger): Int64;
!function NullableBooleanToValue(const V: TNullableBoolean; out Value: Boolean): Boolean;
!...
Those {\f1\fs20 Nullable*ToValue()} functions are mandatory for use under FPC, which does not allow mixing plain {\f1\fs20 variant} values and specialized {\f1\fs20 TNullable* = type variant} values.
Thanks to those types, and their corresponding wrapper functions, you have at hand everything needed to safely store some nullable values into your application database, with proper handling on Delphi side.
:164 Working with Objects
To access a particular record, the following code can be used to handle @*CRUD@ statements ({\i Create Retrieve Update Delete} actions are implemented via {\i Add/Update/Delete/Retrieve} methods), following the @*REST@ful pattern - see @9@, and using the {\f1\fs20 ID} @*primary key@ as resource identifier:
!!procedure Test(Client: TSQLRest);  // we will use CRUD operations on a REST instance
!var Baby: TSQLBaby;   // store a record
!    ID: TID;          // store a reference to a record
!begin
!  // create and save a new record, since Smith, Jr was just born
!  Baby := TSQLBaby.Create;
!  try
!    Baby.Name := 'Smith';
!    Baby.Address := 'New York City';
!    Baby.BirthDate := Date;
!    Baby.Sex := sMale;
!!    ID := Client.Add(Baby,true);
!  finally
!    Baby.Free; // manage memory as usual
!  end;
!  // update record data
!!  Baby := TSQLBaby.Create(Client,ID); // retrieve from ID
!  try
!    assert(Baby.Name='Smith');
!    Baby.Name := 'Smeeth';
!!    Client.Update(Baby);
!  finally
!    Baby.Free;
!  end;
!  // retrieve record data
!  Baby := TSQLBaby.Create;
!  try
!!    Client.Retrieve(ID,Baby);
!    // we may have written:  Baby := TSQLBaby.Create(Client,ID);
!    assert(Baby.Name='Smeeth');
!  finally
!    Baby.Free;
!  end;
!  // delete the created record
!!  Client.Delete(TSQLBaby,ID);
!end;
Of course, you can have a {\f1\fs20 TSQLBaby} instance alive during a longer time. The same {\f1\fs20 TSQLBaby} instance can be used to access several record content, and call {\f1\fs20 Retrieve / Add / Delete / Update} methods on purpose.
No @*SQL@ statement to write, nothing to care about database engine expectations (e.g. for date or numbers processing): just accessing objects via high-level methods. It could even work with @*NoSQL@ databases, like a fast {\f1\fs20 TObjectList} or @*MongoDB@. This is the magic of @*ORM@.
To be honest, the REST pattern does not match directly the CRUD operations exactly. We had to tied a little bit the REST verbs - as defined @9@ - to fit our ORM purpose. But all you have to know is that those {\i Add/Update/Delete/Retrieve} methods are able to define the full persistence lifetime of your precious objects.
: Queries
:  Return a list of objects
You can query your table with the {\f1\fs20 FillPrepare} or {\f1\fs20 @**CreateAndFillPrepare@} methods, for instance all babies with balls and a name starting with the letter 'A':
!var aMale: TSQLBaby;
!...
!!aMale := TSQLBaby.CreateAndFillPrepare(Client,
!!  'Name LIKE ? AND Sex = ?',['A%',ord(sMale)]);
!try
!!  while aMale.FillOne do
!    DoSomethingWith(aMale);
!finally
!  aMale.Free;
!end;
This request loops through all matching records, accessing each row content via a {\f1\fs20 TSQLBaby} instance.
The {\f1\fs20 mORMot} engine will create a SQL statement with the appropriate SELECT query, retrieve all data as JSON, transmit it between the Client and the Server (if any), then convert the values into properties of our {\f1\fs20 TSQLBaby} object instance. Internally, the {\f1\fs20 [CreateAnd]FillPrepare} / {\f1\fs20 FillOne} methods use a list of records, retrieved as @*JSON@ from the Server, and parsed in memory one row a time (using an internal {\f1\fs20 @*TSQLTableJSON@} instance).
Note that there is an optional {\f1\fs20 aCustomFieldsCSV} parameter available in all {\f1\fs20 FillPrepare / CreateAndFillPrepare} methods, by which you may specify a CSV list of field names to be retrieved. It may save some remote bandwidth, if not all record fields values are needed in the loop. Note that you should use this {\f1\fs20 aCustomFieldsCSV} parameter only to retrieve some data, and that the other fields will remain untouched (i.e. void in case of {\f1\fs20 CreateAndFillPrepare}): any later call to {\f1\fs20 Update} should lead into a data loss, since the method will know that is has been called during a {\f1\fs20 FillPrepare / CreateAndFillPrepare} process, and only the retrieved filled will be updated on the server side.
You could also create a {\f1\fs20 TObjectList}, or - even better for newer versions of {\i Delphi} supporting the generics syntax - a {\f1\fs20 TObjectList<T>} instance to retrieve all values of a table:
!var aList: TObjectList<TSQLBaby>;
!    aMale: TSQLBaby;
!...
!!aList := Client.RetrieveList<TSQLBaby>(
!!  'Name LIKE ? AND Sex = ?',['A%',ord(sMale)]);
!try
!!  for aMale in aList do
!     DoSomethingWith(aMale);
!finally
!  aList.Free;
!end;
Note that this method will use more memory and resources than a {\f1\fs20 *FillPrepare} call followed by a {\f1\fs20 while ...FillOne do} loop, since the later will only allocate one instance of the {\f1\fs20 TSQLRecord}, then fill the properties of this single instance directly from the returned JSON content, one at a time. For huge lists, or in multi-threaded environement, it may make a difference.\line But the generics syntax can make cleaner code, or more integrated with your business logic.
:36  Query parameters
For safer and faster database process, the WHERE clause of the request expects some parameters to be specified. They are bound in the {\f1\fs20 ?} appearance order in the WHERE clause of the {\f1\fs20 [CreateAnd]FillPrepare} query method.
Standard simple kind of parameters ({\f1\fs20 @*RawUTF8@, integer, @*double@, @*currency@}..) can be bound directly - as in the sample code above for {\f1\fs20 Name} or {\f1\fs20 Sex} properties. The first parameter will be bound as {\f1\fs20 'A%' RawUTF8} TEXT, and the second as the {\f1\fs20 1} INTEGER value.
Any {\f1\fs20 @*TDateTime@} bound parameter shall better be specified using {\f1\fs20 @*DateToSQL@()}, {\f1\fs20 @*DateTimeToSQL@()} or {\f1\fs20 @*TimeLogToSQL@()} functions, as such:
! aRec.CreateAndFillPrepare(Client,'Datum=?',[DateToSQL(EncodeDate(2012,5,4))]);
! aRec.CreateAndFillPrepare(Client,'Datum>=?',[DateToSQL(2012,5,4)]);
! aRec.CreateAndFillPrepare(Client,'Datum<=?',[DateTimeToSQL(Now)]);
! aRec.CreateAndFillPrepare(Client,'Datum<=?',[TimeLogToSQL(Client.ServerTimestamp)]);
For {\f1\fs20 @*TTimeLog@ / @*TModTime@ / @*TCreateTime@ / @*TUnixTime@ / @*TUnixMSTime@} kind of properties, please use the underlying {\f1\fs20 Int64} value as bound parameter.
As stated previously, @*BLOB@ (i.e. {\f1\fs20 sftBlob} or {\f1\fs20 @*TSQLRawBlob@}) properties are handled separately, via dedicated {\f1\fs20 RetrieveBlob} and {\f1\fs20 UpdateBlob} method calls (or their global {\f1\fs20 RetrieveBlobFields} / {\f1\fs20 UpdateBlobFields} twins). In fact, BLOB data is expected to be potentially big (more than a few MB). But you can specify a small BLOB content using an explicit conversion to the corresponding TEXT format, by calling {\f1\fs20 @*BinToBase64WithMagic@()} overloaded functions when preparing an UPDATE query, or by defining a {\f1\fs20 TByteDynArray} published field instead of {\f1\fs20 TSQLRawBlob}.\line See also {\f1\fs20 @*ForceBlobTransfert@} and {\f1\fs20 ForceBlobTransfertTable[]} properties of {\f1\fs20 TSQLRestClientURI}.
Note that there was a {\i breaking change} about the {\f1\fs20 TSQLRecord.Create / FillPrepare  / CreateAndFillPrepare} and {\f1\fs20 TSQLRest.OneFieldValue / MultiFieldValues} methods: for historical reasons, they expected parameters to be marked as {\f1\fs20 %} in the SQL WHERE clause, and inlined via {\f1\fs20 :(...):} as stated @61@ - since revision 1.17 of the framework, those methods expect parameters marked as {\f1\fs20 ?} and with no {\f1\fs20 :(...):}. Due to this {\i breaking change}, user code review is necessary if you want to upgrade the engine from 1.16 or previous. In all cases, using {\f1\fs20 ?} is less confusing for new users, and more close to the usual way of preparing database queries - e.g. as stated @27@. Both {\f1\fs20 TSQLRestClient.ExecuteFmt / ListFmt} methods are not affected by this change, since they are just wrappers to the {\f1\fs20 FormatUTF8()} function.
For the most complex codes, you may want to prepare ahead the WHERE clause of the ORM request. You may use the overloaded {\f1\fs20 FormatUTF8()} function as such:
!var where: RawUTF8;
!begin
!  where := FormatUTF8('id=?', [], [SomeID]);
!  if add_active then
!    where := FormatUTF8('% and active=?', [where], [ActiveFlag]);
!  if add_date_ini then
!    where := FormatUTF8('% and date_ini>=?', [where], [DateToSQL(Date-2)]);
! ...
Then the request will be easy to create, and fast to execute, thanks to prepared statements in the framework database layer.
:  Introducing TSQLTableJSON
As we stated above, {\f1\fs20 [CreateAnd]FillPrepare} / {\f1\fs20 FillOne} methods are implemented via an internal {\f1\fs20 @**TSQLTableJSON@} instance.
In short, {\f1\fs20 TSQLTableJSON} will expect some {\i @*JSON@} content as input, will parse it in rows and columns, associate it with one or more optional {\f1\fs20 @*TSQLRecord@} class types, then will let you access the data via its {\f1\fs20 Get*} methods.
You can use this {\f1\fs20 TSQLTableJSON} class as in the following example:
!procedure WriteBabiesStartingWith(const Letters: RawUTF8; Sex: TSex);
!var aList: TSQLTableJSON;
!    Row: integer;
!begin
!!  aList := Client.MultiFieldValues(TSQLBaby,'ID,BirthDate',
!!    'Name LIKE ? AND Sex = ?',[Letters+'%',ord(Sex)]);
!  if aList=nil then
!    raise Exception.Create('Impossible to retrieve data from Server');
!  try
!    for Row := 1 to aList.RowCount do
!      writeln('ID=',aList.GetAsInteger(Row,0),' BirthDate=',aList.Get(Row,1));
!  finally
!    aList.Free;
!  end;
!end;
For a record with a huge number of fields, specifying the needed fields could save some bandwidth. In the above sample code, the {\f1\fs20 ID} column has a field index of 0 (so is retrieved via {\f1\fs20 aList.GetAsInteger(Row,0)}) and the {\f1\fs20 BirthDate} column has a field index of 1 (so is retrieved as a {\f1\fs20 PUTF8Char} via {\f1\fs20 aList.Get(Row,1)}). All data rows are processed via a loop using the {\f1\fs20 RowCount} property count - first data row is indexed as 1, since the row 0 will contain the column names.
The {\f1\fs20 TSQLTable} class has some methods dedicated to direct cursor handling, as such:
!procedure WriteBabiesStartingWith(const Letters: RawUTF8; Sex: TSex);
!var aList: TSQLTableJSON;
!begin
!  aList := Client.MultiFieldValues(TSQLBaby,'ID,BirthDate',
!    'Name LIKE ? AND Sex = ?',[Letters+'%',ord(Sex)]);
!  try
!!    while aList.Step do
!!      writeln('ID=',aList.Field(0),' BirthDate=',aList.Field(1));
!  finally
!    aList.Free;
!  end;
!end;
By using the {\f1\fs20 TSQLTable.Step} method, you do not need to check that {\f1\fs20 aList<>nil}, since it will return false if {\f1\fs20 aList} is not assigned. And you do not need to access the {\f1\fs20 RowCount} property, nor specify the current row number.
We may have used not the field index, but the field name, within the loop:
!      writeln('ID=',aList.Field('ID'),' BirthDate=',aList.Field('BirthDate'));
You can also access the field values using @*late-binding@ and a local {\f1\fs20 variant}, which gives some perfectly readable code:
!procedure WriteBabiesStartingWith(const Letters: RawUTF8; Sex: TSex);
!var baby: variant;
!begin
!  with Client.MultiFieldValues(TSQLBaby,'ID,BirthDate',
!    'Name LIKE ? AND Sex = ?',[Letters+'%',ord(Sex)]) do
!  try
!!    while Step(false,@baby) do
!!      writeln('ID=',baby.ID,' BirthDate=',baby.BirthDate);
!  finally
!    Free;
!  end;
!end;
In the above code, late-binding will search for the {\f1\fs20 "ID"} and {\f1\fs20 "BirthDate"} fields at runtime. But the ability to write {\f1\fs20 baby.ID} and {\f1\fs20 baby.BirthDate} is very readable. Using a {\f1\fs20 with ... do} statement makes the code shorter, but should be avoided if it leads into confusion, e.g. in case of more complex process within the loop.
See also the following methods of {\f1\fs20 @*TSQLRest@}: {\f1\fs20 OneFieldValue}, {\f1\fs20 OneFieldValues}, {\f1\fs20 MultiFieldValue}, {\f1\fs20 MultiFieldValues} which are able to retrieve either a {\f1\fs20 TSQLTableJSON}, or a {\i @*dynamic array@} of {\f1\fs20 integer} or {\f1\fs20 @*RawUTF8@}. And also {\f1\fs20 List} and {\f1\fs20 ListFmt} methods of {\f1\fs20 TSQLRestClient}, if you want to make a {\f1\fs20 @*JOIN@} against multiple tables at once.
A {\f1\fs20 @*TSQLTableJSON@} content can be associated to a {\f1\fs20 TGrid} in order to produce an User Interface taking advantage of the column types, as retrieved from the associated {\f1\fs20 @*TSQLRecord@} @*RTTI@. The {\f1\fs20 TSQLTableToGrid} class is able to associate any {\f1\fs20 TSQLTable} to a standard {\f1\fs20 TDrawGrid}, with some enhancements: themed drawing, handle Unicode, column types (e.g. {\f1\fs20 boolean} are displayed as check-boxes, dates as text, etc...), column auto size, column sort, incremental key lookup, optional hide IDs, selection...
:61  Note about query parameters
{\i (this paragraph is not mandatory to be read at first, so you can skip it if you do not need to know about the mORMot internals - just remember that ? bound parameters are inlined as {\f1\fs20 :(...):} in the JSON transmitted content so can be set directly as such in any WHERE clause)}
If you consider the first sample code:
!aMale := TSQLBaby.CreateAndFillPrepare(Client,
!  'Name LIKE ? AND Sex = ?',['A%',ord(sMale)]);
This will execute a SQL statement, with an ORM-generated SELECT, and a WHERE clause using two parameters bound at execution, containing {\f1\fs20 'A%' RawUTF8} text and 1 integer value.
In fact, from the SQL point of view, the {\f1\fs20 CreateAndFillPrepare()} method as called here is exactly the same as:
!aMale := TSQLBaby.CreateAndFillPrepare(Client,
!  'Name LIKE :(''A%''): AND Sex = :(1):');
or
!aMale := TSQLBaby.CreateAndFillPrepare(Client,
!  'Name LIKE :(%): AND Sex = :(%):',['''A%''',ord(sMale)],[]));
or
!aMale := TSQLBaby.CreateAndFillPrepare(Client,
!  FormatUTF8('Name LIKE :(%): AND Sex = :(%):',['''A%''',ord(sMale)]));
First point is that the {\f1\fs20 'A'} letter has been embraced with @*quotes@, as expected per the @*SQL@ syntax. In fact, {\f1\fs20 Name LIKE :(%): AND Sex = :(%):', ['''A%''',ord(sMale)]} is expected to be a valid WHERE clause of a SQL statement.
Note we used single quotes, but we may have used double quotes (") inside the {\f1\fs20 :( ):} statements. In fact, {\i @*SQLite3@} expects single @**quotes@ in its raw SQL statements, whereas our @**prepared@ statements {\f1\fs20 :( ):} will handle both single ' and double " quotes. Just to avoid any confusion, we'll always show single quotes in the documentation. But you can safely use double quotes within {\f1\fs20 :( ):} statements, which could be more convenient than single quotes, which should be doubled within a pascal constant string {\f1\fs20 ''}.
The only not-obvious syntax in the above code is the {\f1\fs20 :(%):} used for defining prepared parameters in the format string.
In fact, the format string will produce the following WHERE clause parameter as plain text:
!aMale := TSQLBaby.CreateAndFillPrepare(Client,
!  'Name LIKE :(''A%''): AND Sex = :(1):');
So that the following SQL query will be executed by the database engine, after translation by the @*ORM@ magic:
$ SELECT * FROM Baby WHERE Name LIKE ? AND Sex = ?;
With the first {\f1\fs20 ?} parameter bound with {\f1\fs20 'A%'} value, and the second with {\f1\fs20 1}.
In fact, when the framework finds some {\f1\fs20 :( ):} in the SQL statement string, it will prepare a SQL statement, and will bound the parameters before execution (in our case, text {\f1\fs20 A} and integer {\f1\fs20 1}), reusing any previous matching prepared SQL statement. See @14@ for more details about this mechanism.
To be clear, without any prepared statement, you could have used:
!aMale := TSQLBaby.CreateAndFillPrepare(Client,
!  'Name LIKE % AND Sex = %',['''A%''',ord(sMale)],[]);
or
!aMale := TSQLBaby.CreateAndFillPrepare(Client,
!  FormatUTF8('Name LIKE % AND Sex = %',['''A%''',ord(sMale)]));
which will produce the same as:
!aMale := TSQLBaby.CreateAndFillPrepare(Client,
!  'Name LIKE ''A%'' AND Sex = 1');
So that the following SQL statement will be executed:
$ SELECT * FROM Baby WHERE Name LIKE 'A%' AND Sex = 1;
Note that we prepared the SQL WHERE clause, so that we could use the same request statement for all females with name starting with the character 'D':
!aFemale := TSQLBaby.CreateAndFillPrepare(Client,
!  'Name LIKE :(%): AND Sex = :(%):', ['''D%''',ord(sFemale)]);
Using a prepared statement will speed up the database engine, because the SQL query will have to be parsed and optimized only once.
The second query method, i.e.
!  aList := Client.MultiFieldValues(TSQLBaby,'ID,BirthDate',
!    'Name LIKE ? AND Sex = ?',[Letters+'%',ord(Sex)]);
is the same as this code:
!  aList := Client.MultiFieldValues(TSQLBaby,'ID,BirthDate',
!    'Name LIKE :(%): AND Sex = :(%):',[QuotedStr(Letters+'%'),ord(Sex)],[]);
or
!  aList := Client.MultiFieldValues(TSQLBaby,'ID,BirthDate',
!    FormatUTF8('Name LIKE :(%): AND Sex = :(%):',[QuotedStr(Letters+'%'),ord(Sex)]));
In both cases, the parameters will be inlined, in order to prepare the statements, and improve execution speed.
We used the {\f1\fs20 QuotedStr} standard function to embrace the {\f1\fs20 Letters} parameter with quotes, as expected per the @*SQL@ syntax.
Of course, using '?' and bounds parameters is much easier than '%' and manual {\f1\fs20 :(%):} in-lining with a {\f1\fs20 QuotedStr()} function call. In your client code, you should better use '?' - but if you find some {\f1\fs20 ':(%):'} in the framework source code and when a WHERE clause is expected within the transmitted JSON content, you won't be surprised.
:130 Automatic TSQLRecord memory handling
Working with objects is pretty powerful, but requires to handle manually the created instances life time, via {\f1\fs20 try} .. {\f1\fs20 finally} blocks. Most of the time, the {\f1\fs20 TSQLRecord} life time will be very short: we allocate one instance on a local variable, then release it when it goes out of scope.
If we take again the {\f1\fs20 TSQLBaby} sample, we may write:
!!function NewMaleBaby(Client: TSQLRest; const Name,Address: RawUTF8): TID;
!var Baby: TSQLBaby;   // store a record
!begin
!  Baby := TSQLBaby.Create;
!  try
!    Baby.Name := Name;
!    Baby.Address := Address;
!    Baby.BirthDate := Date;
!    Baby.Sex := sMale;
!!    result := Client.Add(Baby,true);
!  finally
!    Baby.Free;
!  end;
!end;
To ease this pretty usual pattern, the framework offers some kind of automatic memory management at {\f1\fs20 TSQLRecord} level:
!!function NewMaleBaby(Client: TSQLRest; const Name,Address: RawUTF8): TID;
!var Baby: TSQLBaby;   // store a record
!begin
!!  TSQLBaby.AutoFree(Baby);  // no try..finally needed!
!  Baby.Name := Name;
!  Baby.Address := Address;
!  Baby.BirthDate := Date;
!  Baby.Sex := sMale;
!!  result := Client.Add(Baby,true);
!end; // local Baby instance will be released here
It may also be useful for queries.\line Instead of writing:
!var aMale: TSQLBaby;
!...
!  aMale := TSQLBaby.CreateAndFillPrepare(Client,
!    'Name LIKE ? AND Sex = ?',['A%',ord(sMale)]);
!  try
!    while aMale.FillOne do
!      DoSomethingWith(aMale);
!  finally
!    aMale.Free;
!  end;
We may write:
!var aMale: TSQLBaby;
!...
!  TSQLBaby.AutoFree(aMale,Client,'Name LIKE ? AND Sex = ?',['A%',ord(sMale)]);
!  while aMale.FillOne do
!    DoSomethingWith(aMale);
Without the need to write the {\f1\fs20 try ... finally} block.
See the {\f1\fs20 TSQLRecord.AutoFree()} overloaded methods in {\f1\fs20 mORMot.pas} for the several use cases, and the associated {\f1\fs20 TAutoFree} / {\f1\fs20 IAutoFree} types as defined in {\f1\fs20 SynCommons.pas}. Note that you can handle several local variables in a single {\f1\fs20 TSQLRecord.AutoFree()} or {\f1\fs20 TAutoFree.Create()} initialization.
Be aware that it does not introduce some kind of magic @*garbage collector@, as available in C# or Java. It is not even similar to the {\f1\fs20 ARC} memory model used by {\i Apple} and the {\i Delphi} {\i NextGen} compiler. It is just some syntaxic sugar creating a local hidden {\f1\fs20 IAutoFree} interface, which will be released at the end of the local method by the compiler, and also release all associated class instances. So the local class instances should stay in the local scope, and should not be sent and stored in another process: in such cases, you may encounter access violation issues.
Due to an issue (feature?) in the @*FPC@ implementation of interfaces - see @http://bugs.freepascal.org/view.php?id=26602 - the above code will not work directly. You should assign the result of this method to a local {\f1\fs20 IAutoFree} variable, as such:
!var aMale: TSQLBaby;
!!    auto: IAutoFree;
!...
!  auto := TSQLBaby.AutoFree(aMale,Client,'Name LIKE ? AND Sex = ?',['A%',ord(sMale)]);
!  while aMale.FillOne do
!    DoSomethingWith(aMale);
One alternative may be to use a {\f1\fs20 with} statement, which prevents the need of defining a local variable:
!var aMale: TSQLBaby;
!...
!  with TAutoFree.One(aMale,TSQLBaby.CreateAndFillPrepare(Client,
!    'Name LIKE ? AND Sex = ?',['A%',ord(sMale)])) do
!    while aMale.FillOne do
!      DoSomethingWith(aMale);
Or use one of the {\f1\fs20 TSQLRecord.AutoFree} overloaded {\f1\fs20 class} methods:
!var aMale: TSQLBaby;
!...
!  with TSQLBaby.AutoFree(aMale,Client,'Name LIKE ? AND Sex = ?',['A%',ord(sMale)]) do
!    while aMale.FillOne do
!      DoSomethingWith(aMale);
If you want your code to cross-compile with both Delphi and FPC, consider this expectation of the FPC compiler.
: Objects relationship: cardinality
All previous code is fine if your application requires "flat" data. But most of the time, you'll need to define master/child relationship, perhaps over several levels. In data modeling, the {\i @**cardinality@} of one data table with respect to another data table is a critical aspect of database design. Relationships between data tables define {\i cardinality} when explaining how each table links to another.
In the relational model, tables can have the following {\i cardinality}, i.e. can be related as any of:
- "@*One to one@".
- "Many to one" (rev. "@*One to many@");
- "Many to many" (or "@has many@").
Our {\i mORMot framework} handles all those kinds of {\i cardinality}.
:70  "One to one" or "One to many"
:   TSQLRecord published properties are IDs, not instance
In order to handle "{\i @**One to one@}" or "{\i @**One to many@}" relationship between tables (i.e. normalized @**Master/Detail@ in a classical @*RDBMS@ approach), you could define {\f1\fs20 @*TSQLRecord@} @*published properties@ in the object definition.
For instance, you could declare classes as such:
!  TSQLMyFileInfo = class(TSQLRecord)
!  private
!    FMyFileDate: TDateTime;
!    FMyFileSize: Int64;
!  published
!    property MyFileDate: TDateTime read FMyFileDate write FMyFileDate;
!    property MyFileSize: Int64 read FMyFileSize write FMyFileSize;
!  end;
!
!  TSQLMyFile = class(TSQLRecord)
!  private
!    FSecondOne: TSQLMyFileInfo;
!    FFirstOne: TSQLMyFileInfo;
!    FMyFileName: RawUTF8;
!  published
!    property MyFileName: RawUTF8 read FMyFileName write FMyFileName;
!!    property FirstOne: TSQLMyFileInfo read FFirstOne write FFirstOne;
!!    property SecondOne: TSQLMyFileInfo read FSecondOne write FSecondOne;
!  end;
As stated by @26@, {\f1\fs20 TSQLRecord} published properties do not contain an instance of the {\f1\fs20 TSQLRecord} class. They will instead contain {\f1\fs20 pointer(RowID)}, and will be stored as an {\f1\fs20 INTEGER} in the database.
So the main rule is to {\i never use directly such published properties}, as if they were regular class instance: otherwise you'll have an unexpected {\i access violation} error.
:   Transtyping IDs
When creating such records, use temporary instances for each detail object, as such:
!var One, Two: TSQLMyFileInfo;
!     MyFile: TSQLMyFile;
!begin
!  One := TSQLMyFileInfo.Create;
!  Two := TSQLMyFileInfo.Create;
!  MyFile := TSQLMyFile.Create;
!  try
!    One.MyFileDate := ....
!    One.MyFileSize := ...
!!    MyFile.FirstOne := TSQLMyFileInfo(MyDataBase.Add(One,True)); // add One and store ID in MyFile.FirstOne
!    Two.MyFileDate := ....
!    Two.MyFileSize := ...
!!    MyFile.SecondOne:= TSQLMyFileInfo(MyDataBase.Add(Two,True)); // add Two and store ID in MyFile.SecondOne
!    MyDataBase.Add(MyFile,true);
!  finally
!     MyFile.Free;
!     Two.Free;
!     One.Free;
!  end;
!end;
Note that you those two assignments are the same:
! MyFile.FirstOne := TSQLMyFileInfo(MyDataBase.Add(One,True));
! MyFile.FirstOne := pointer(MyDataBase.Add(One,True));
Or you may have added the {\f1\fs20 One} row first:
!  MyDatabase.Add(One,true);
then assigned it to the {\f1\fs20 MyFile} record on one of the following expressions:
! MyFile.FirstOne := TSQLMyFileInfo(One.ID);
! MyFile.FirstOne := pointer(One.ID);
!! MyFile.FirstOne := One.AsTSQLRecord;
The first two statements, using a {\f1\fs20 class/pointer} type cast will work only in 32-bit (since ID is an integer). Using {\f1\fs20 TSQLRecord.@*AsTSQLRecord@} property will work on all platforms, including 64-bit, and is perhaps easier to deal with in your code.
When accessing the detail objects, you should not access directly to {\f1\fs20 FirstOne} or {\f1\fs20 SecondOne} properties (there are not class instances, but integer IDs), then use instead the {\f1\fs20 TSQLRecord. Create(aClient: TSQLRest; aPublishedRecord: TSQLRecord: ForUpdate: boolean=false)} overloaded constructor, as such:
!var One: TSQLMyFileInfo;
!    MyFile: TSQLMyFile;
!begin
!  MyFile := TSQLMyFile.Create(Client,aMyFileID);
!  try
!    // here MyFile.FirstOne.MyFileDate will trigger an access violation
!!    One := TSQLMyFileInfo.Create(Client,MyFile.FirstOne);
!    try
!      // here you can access One.MyFileDate or One.MyFileSize
!    finally
!      One.Free;
!    end;
!  finally
!    MyFile.Free;
!  end;
!end;
Or with a {\f1\fs20 with} statement:
!var MyFile: TSQLMyFile;
!begin
!  MyFile := TSQLMyFile.Create(Client,aMyFileID);
!  try
!    // here MyFile.FirstOne.MyFileDate will trigger an access violation
!    with TSQLMyFileInfo.Create(Client,MyFile.FirstOne) do
!    try
!      // here you can access MyFileDate or MyFileSize
!    finally
!      Free;
!    end;
!  finally
!    MyFile.Free;
!  end;
!end;
Mapping a {\f1\fs20 TSQLRecord} field into an {\f1\fs20 integer} ID is a bit difficult to learn at first. It was the only way we found out in order to define a "one to one" or "one to many" relationship within the class definition, without any property attribute features of the {\i Delphi} compiler (only introduced in newer versions). The main drawback is that the compiler won't be able to identify at compile time some potential GPF issues at run time. This is up to the developer to write correct code, when dealing with {\f1\fs20 TSQLRecord} properties. Using {\f1\fs20 AsTSQLRecord} property and overloaded {\f1\fs20 TSQLRecord. Create(aPublishedRecord)} constructor will help a lot.
:129   Automatic instantiation and JOINed query
Having to manage at hand all nested {\f1\fs20 TSQLRecord} instances can be annoying, and error-prone.
As an alternative, if you want to retrieve a whole {\f1\fs20 TSQLRecord} instance including its nested {\f1\fs20 TSQLRecord} published properties, you can use either of those two constructors:
- {\f1\fs20 TSQLRecord.@**CreateJoined@(aClient,aID)};
- {\f1\fs20 TSQLRecord.CreateAndFillPrepareJoined()}, followed by a {\f1\fs20 while FillOne do ....} loop.
Both constructors:
- Will {\i auto-instantiate} all {\f1\fs20 TSQLRecord} published properties;
- Then the ORM core will retrieve all properties, included nested {\f1\fs20 TSQLRecord} via a {\f1\fs20 SELECT .... LEFT JOIN ...} statement;
- Then the nested {\f1\fs20 TSQLRecord} will be released at {\f1\fs20 Destroy} of the main instance (to avoid any unexpected memory leak).
So you can safely write:
!var MyFile: TSQLMyFile;
!begin
!!  MyFile := TSQLMyFile.CreateJoined(Client,aMyFileID);
!  try
!    // here MyFile.FirstOne and MyFile.SecondOne are true instances
!    // and have already retrieved from the database by the constructor
!    // so you can safely access MyFile.FirstOne.MyFileDate or MyFile.SecondOne.MyFileSize here!
!  finally
!    MyFile.Free; // will release also MyFile.FirstOne and MyFile.SecondOne
!  end;
!end;
Note that this will work as expected when retrieving some data from the database, but, in the current implementation of the ORM, any {\f1\fs20 Update()} call will manage only the main {\f1\fs20 TSQLRecord} properties, and the nested {\f1\fs20 TSQLRecord} properties {\f1\fs20 ID}, not the nested properties values. For instance, in code above, {\f1\fs20 aClient.Update(MyFile)} will update the {\f1\fs20 TSQLMyFile} table, but won't reflect any modification to {\f1\fs20 MyFile.FirstOne} or {\f1\fs20 MyFile.SecondOne} properties. This limitation may be removed in the future - you may ask explicitly for this feature request.
:  "Has many" and "has many through"
As @http://en.wikipedia.org/wiki/Many-to-many_(data_model) wrote:
{\i In systems analysis, a many-to-many relationship is a type of cardinality that refers to the relationship between two entities (see also Entity-Relationship Model) A and B in which A may contain a parent row for which there are many children in B and vice versa. For instance, think of A as Authors, and B as Books. An Author can write several Books, and a Book can be written by several Authors. Because most database management systems only support one-to-many relationships, it is necessary to implement such relationships physically via a third and fourth junction table, say, AB with two one-to-many relationships A -> AB and B -> AB. In this case the logical @*primary key@ for AB is formed from the two foreign keys (i.e. copies of the primary keys of A and B).}
From the record point of view, and to follow the @*ORM@ vocabulary (in Ruby on Rails, Python, or other {\i ActiveRecord} clones), we could speak of "@**has many@" relationship. In the classic RDBMS implementation, a pivot table is created, containing two references to both related records. Additional information can be stored within this pivot table. It could be used, for instance, to store association time or corresponding permissions of the relationship. This is called a "@**has many through@" relationship.
In fact, there are several families of ORM design, when implementing the "many to many" @*cardinality@:
- Map collections into {\f1\fs20 @*JOIN@}ed query from the ORM (i.e. pivot tables are abstracted from object lists or collections by the framework, to implement "has many" relationship, but you will have to define @*lazy loading@ and won't have "has many through" relationship at hand);
- Explicitly handle pivot tables as ORM classes, and provide methods to access to them (it will allow both "has many" and "has many through" relationship).
- Store collections within the ORM classes property (data @*sharding@).
In the {\i mORMot framework}, we did not implement the 1st implementation pattern, but the 2nd and 3rd:
- You can map the DB with dedicated {\f1\fs20 @*TSQLRecordMany@} classes, which allows some true pivot table to be available (that is the 2nd family), introducing true "has many through" cardinality;
- But for most applications, it sounds definitively more easy to use {\f1\fs20 TCollection} (of {\f1\fs20 TPersistent} classes) or {\i dynamic arrays} within one {\f1\fs20 TSQLRecord} class, and data sharding (i.e. the 3rd family).
Up to now, there is no explicit {\i @**Lazy Loading@} feature in our ORM. There is no native handling of {\f1\fs20 @*TSQLRecord@} collections or lists (as they do appear in the first family of ORMs). This could sound like a limitation, but it allows to manage exactly the data to be retrieved from the server in your code, and maintain bandwidth and memory use as low as possible. Use of a pivot table (via the {\f1\fs20 @*TSQLRecordMany@} kind of records) allows tuned access to the data, and implements optimal {\i lazy loading} feature. Note that the only case when some {\f1\fs20 TSQLRecord} instances are automatically created by the ORM is for those {\f1\fs20 TSQLRecordMany} published properties.
:29   Shared nothing architecture (or sharding)
:    Embedding all needed data within the record
Defining a pivot table is a classic and powerful use of relational database, and unleash its power (especially when linked data is huge).
But it is not easy nor natural to properly handle it, since it introduces some dependencies from the DB layer into the business model. For instance, it does introduce some additional requirements, like constraints / integrity checking and tables/classes inter-dependency.
Furthermore, in real life, we do not have such a separated storage, but we store all details within the main data. So for a @54@, which tries to map the real objects of its own domain, such a pivot table is breaking the business logic. With today's computer power, we can safely implement a centralized way of storing data into our data repository.
Let us quote what {\i wikipedia} states at @http://en.wikipedia.org/wiki/Shared_nothing_architecture
{\i A @**shared nothing architecture@ (SN) is a distributed computing architecture in which each node is independent and self-sufficient, and there is no single point of contention across the system. People typically contrast SN with systems that keep a large amount of centrally-stored state information, whether in a database, an application server, or any other similar single point of contention.}
As we stated in @26@, in our ORM, high-level types like @*dynamic array@s or {\f1\fs20 @**TPersistent@} / {\f1\fs20 @**TCollection@} properties are stored as BLOB or TEXT inside the main data row. There is no external linked table, no {\i @*Master/Detail@} to maintain. In fact, each {\f1\fs20 @*TSQLRecord@} instance content could be made self-contained in our ORM.
In particular, you may consider using our @80@ stored in a {\f1\fs20 variant} published property. It will allow to store any complex document, of nested objects or objects. They will be efficiently stored and transmitted as @*JSON@.
When the server starts to have an increasing number of clients, such a data layout could be a major benefit. In fact, the so-called {\i @**sharding@}, or horizontal partitioning of data, is a proven solution for web-scale databases, such as those in use by social networking sites. How does {\i EBay} or {\i Facebook} scale with so many users? Just by {\i sharding}.
A simple but very efficient {\i sharding} mechanism could therefore be implemented with our ORM. In-memory databases, or {\i SQLite3} are good candidate for light speed data process. Even {\i SQLite} could scale very well in most cases, when properly used - see @59@.
Storing detailed data in BLOB or in TEXT as JSON could first sounds a wrong idea. It does break one widely accepted principle of the @*RDBMS@ architecture. But even {\i Google} had to break this dogma. And when {\i @*MySQL@} or any similar widely used databases try to implement sharding, it does need a lot of effort. Others, like the @*NoSQL@ {\i @*MongoDB@}, are better candidates: they are not tight to the SQL/RDBMS flat scheme.
Finally, this implementation pattern fits much better with a {\i @*Domain-Driven@} design. See @68@.
Therefore, on second thought, having at hand a {\i shared nothing architecture} could be a great advantage. Our ORM is already ready to break the table-oriented of SQL. Let us go one step further.
:    Nesting objects and arrays
The "{\i has many}" and "{\i has many through}" relationship we just described does follow the classic process of rows association in a relational database, using a pivot table. This does make sense if you have some DB background, but it is sometimes not worth it.
One drawback of this approach is that the data is split into several tables, and you should carefully take care of data integrity to ensure for instance that when you delete a record, all references to it are also deleted in the associated tables. Our @*ORM@ engine will take care of it, but could fail sometimes, especially if you play directly with the tables via SQL, instead of using high-level methods like {\f1\fs20 FillMany*} or {\f1\fs20 DestGetJoined}.
Another potential issue is that one business logical unit is split into several tables, therefore into several diverse {\f1\fs20 @*TSQLRecord@} and {\f1\fs20 @*TSQLRecordMany@} classes. From the @*ORM@ point of view, this could be confusing.
Starting with the revision 1.13 of the framework, {\i @*dynamic array@s}, {\f1\fs20 @*TStrings@} and {\f1\fs20 @*TCollection@} can be used as @*published properties@ in the {\f1\fs20 TSQLRecord} class definition. This won't be strong enough to implement all possible "Has many" architectures, but could be used in most case, when you need to add a list of records within a particular record, and when this list won't have to be referenced as a stand-alone table.
{\i @*Dynamic array@s} will be stored as @*BLOB@ fields in the database, retrieved with {\i @*Base64@} encoding in the @*JSON@ transmitted stream, then serialized using the {\f1\fs20 TDynArray} wrapper. Therefore, only {\i Delphi} clients will be able to use this field content: you'll loose the @*AJAX@ capability of the ORM, at the benefit of better integration with object pascal code. Some dedicated SQL functions have been added to the {\i SQLite} engine, like {\f1\fs20 @*IntegerDynArrayContains@}, to search inside this @*BLOB@ field content from the WHERE clause of any search (see @21@). Those functions are available from AJAX queries.
{\f1\fs20 @*TPersistent@ / @*TStrings@} and {\f1\fs20 @*TCollection@ / @*TObjectList@} will be stored as TEXT fields in the database, following the {\f1\fs20 ObjectToJSON} function format: you can even serialize any {\f1\fs20 @*TObject@} class, via a previous call to the {\f1\fs20 TJSONSerializer. @*RegisterCustomSerializer@} class method - see @52@ - or {\f1\fs20 @*TObjectList@} list of instances, if they are previously registered by {\f1\fs20 TJSONSerializer. @*RegisterClassForJSON@} - see @71@. This format contains only valid JSON arrays or objects: so it could be un-serialized via an AJAX application, for instance.
About this (trolling?) subject, and why/when you should use plain {\i Delphi} objects or arrays instead of classic @*Master/Detail@ DB relationship, please read "{\i Objects, not tables}" and "{\i ORM is not DB}" paragraphs below.
:     TDocVariant and variant fields
:      Schemaless storage via a variant
As we just wrote, a first-class candidate for {\i data sharding} in a {\f1\fs20 TSQLRecord} is our @80@.
You may define:
! TSQLRecordData = class(TSQLRecord)
! private
!    fName: RawUTF8;
!    fData: variant;
! public
! published
!   property Name: RawUTF8 read fTest write fTest stored AS_UNIQUE;
!!   property Data: variant read fData write fData;
! end;
Here, we defined two indexed keys, ready to access any data record:
- Via the {\f1\fs20 ID: @*TID@} property defined at {\f1\fs20 TSQLRecord} level, which will map the {\i SQLite3} {\f1\fs20 RowID} @*primary key@;
- Via the {\f1\fs20 Name: RawUTF8} property, which will was marked to be indexed by setting the "{\f1\fs20 stored @*AS_UNIQUE@}" attribute.
Then, any kind of data may be stored in the {\f1\fs20 Data: variant} published property. In the database, it will be stored as @*JSON@ @*UTF-8@ text, ready to be retrieved from any client, including AJAX / HTML5 applications. {\i Delphi} clients or servers will access those data via {\i @*late-binding@}, from its {\f1\fs20 TDocVariant} instance.
You just reproduced the {\i @**schema-less@} approach of the @*NoSQL@ database engines, in a few lines of code! Thanks to the {\i mORMot}'s @6@ design, your applications are able to store any kind of document, and easily access to them via HTTP.
The documents stored in such a database can have varying sets of fields, with different types for each field.  One could have the following objects in a single collection of our {\f1\fs20 Data: variant} rows:
! { name : "Joe", x : 3.3, y : [1,2,3] }
! { name : "Kate", x : "abc" }
! { q : 456 }
Of course, when using the database for real problems, the data does have a fairly consistent structure. Something like the following will be more common, e.g. for a table persisting {\i student} objects:
! { name : "Joe", age : 30, interests : "football" }
! { name : "Kate", age : 25 }
Generally, there is a direct analogy between this {\i schema-less} style and dynamically typed languages. Constructs such as those above are easy to represent in {\i PHP}, {\i Python} and {\i Ruby}. And, thanks to our {\f1\fs20 TDocVariant} {\i late-binding} magic, even our good {\i Delphi} is able to handle those structures in our code. What we are trying to do here is make this mapping to the database natural, like:
!var aRec: TSQLRecordData;
!    aID: TID;
!begin
!  // initialization of one record
!  aRec := TSQLRecordData.Create;
!  aRec.Name := 'Joe';                              // one unique key
!  aRec.data := _JSONFast('{name:"Joe",age:30}');   // create a TDocVariant
!  // or we can use this overloaded constructor for simple fields
!  aRec := TSQLRecordData.Create(['Joe',_ObjFast(['name','Joe','age',30])]);
!  // now we can play with the data, e.g. via late-binding:
!  writeln(aRec.Name);     // will write 'Joe'
!  writeln(aRec.Data);     // will write '{"name":"Joe","age":30}' (auto-converted to JSON string)
!  aRec.Data.age := aRec.Data.age+1;    // one year older
!  aRec.Data.interests := 'football';   // add a property to the schema
!  aID := aClient.Add(aRec,true);       // will store {"name":"Joe","age":31,"interests":"footbal"}
!  aRec.Free;
!  // now we can retrieve the data either via the aID created integer, or via Name='Joe'
!end;
One of the great benefits of these dynamic objects is that schema migrations become very easy. With a traditional RDBMS, releases of code might contain data migration scripts. Further, each release should have a reverse migration script in case a rollback is necessary. {\f1\fs20 ALTER TABLE} operations can be very slow and result in scheduled downtime.
With a {\i schema-less} organization of the data, 90% of the time adjustments to the database become transparent and automatic. For example, if we wish to add GPA to the {\i student} objects, we add the attribute, re-save, and all is well - if we look up an existing student and reference GPA, we just get back null. Further, if we roll back our code, the new GPA fields in the existing objects are unlikely to cause problems if our code was well written.
In fact, {\i SQlite3} is so efficient about its indexes B-TREE storage, that such a structure may be used as a credible alternative to much heavier {\i NoSQL} engines, like {\i MongoDB} or {\i CouchDB}.\line With the possibility to add some "regular" fields, e.g. plain numbers (like ahead-computed @*aggregation@ values), or text (like a summary or description field), you can still use any needed fast SQL query, without the complexity of {\i @*map/reduce@} algorithm used by the {\i NoSQL} paradigm. You could even use the {\i @*Full Text@ Search} - FTS3/FTS4/FTS5, see @8@ - or @*RTREE@ extension advanced features of {\i SQLite3} to perform your queries. Then, thanks to {\i mORMot}'s ability to access any external database engine, you are able to perform a JOINed query of your {\i schema-less} data with some data stored e.g. in an @*Oracle@, @*PostgreSQL@ or @*MS SQL@ enterprise database. Or switch later to a true {\i @*MongoDB@} storage, in just one line of code - see @84@.
:      JSON operations from SQL code
As we stated, any {\f1\fs20 variant} field will be serialized as @*JSON@, then stored as plain TEXT in the database. In order to make a complex query on the stored JSON, you could retrieve it in your end-user code, then use the corresponding {\f1\fs20 @*TDocVariant@} instance to perform the search on its content. Of course, all this has a noticeable performance cost, especially when the data tend to grow.
The natural way of solving those performance issue is to add some "regular" RDBMS fields, with a proper index, then perform the requests on those fields. But sometimes, you may need to do some addition query, perhaps in conjunction with "regular" field lookup, on the JSON data stored itself.\line In order to avoid the slowest conversion to the ORM client side, we defined some @*SQL function@s, dedicated to JSON process.
The first is {\f1\fs20 JsonGet()}, and is able to extract any value from the TEXT field, mapping a {\f1\fs20 variant}:
|%47%53
|{\f1\fs20 JsonGet(ArrColumn,0)}|returns a property value by index, from a JSON array
|{\f1\fs20 JsonGet(ObjColumn,'PropName')}|returns a property value by name, from a JSON object
|{\f1\fs20 JsonGet(ObjColumn,'Obj1.Obj2.Prop')}|returns a property value by path, including nested JSON objects
|{\f1\fs20 JsonGet(ObjColumn,'Prop1,Prop2')}|extract properties by name, from a JSON object
|{\f1\fs20 JsonGet(ObjColumn,'Prop1,Obj1.Prop')}|extract properties by name (including nested JSON objects), from a JSON object
|{\f1\fs20 JsonGet(ObjColumn,'Prop*')}|extract properties by wildchar name, from a JSON object
|{\f1\fs20 JsonGet(ObjColumn,'Prop*,Obj1.P*')}|extract properties by wildchar name (including nested JSON objects), from a JSON object
|%
If no value does match, this function will return the SQL {\f1\fs20 NULL}. If the matching value is a simple JSON text or number, it will be returned as a TEXT, INTEGER or DOUBLE value, ready to be passed as a result column or any WHERE clause. If the returned value is a nested JSON object or array, it will be returned as TEXT, serialized as JSON; as a consequence, you may use it as the source of another {\f1\fs20 JsonGet()} function, or even able to gather the results via the {\f1\fs20 CONCAT()} aggregate function.
The comma-separated syntax allowed in the property name parameter (e.g. {\f1\fs20 'Prop1,Prop2,Prop3'}), will search for several properties at once in a single object, returning a JSON object of all matching values - e.g. {\f1\fs20 '\{"Prop2":"Value2","Prop3":123\}'} if the {\f1\fs20 Prop1} property did not appear in the stored JSON object.
If you end the property name with a {\f1\fs20 *} character, it will return a JSON object, with all matching properties. Any nested object will have its property names be flattened as {\f1\fs20 \{"Obj1.Prop":...\}}, within the returned JSON object.\line Note that the comma-separated syntax also allows such wildchar search, so that e.g.
$ JsonGet(ObjColumn,'owner') = {"login":"smith","id":123456} as TEXT
$ JsonGet(ObjColumn,'owner.login') = "smith" as TEXT
$ JsonGet(ObjColumn,'owner.id') = 123456 as INTEGER
$ JsonGet(ObjColumn,'owner.name') = NULL
$ JsonGet(ObjColumn,'owner.login,owner.id') = {"owner.login":"smith","owner.id":123456} as TEXT
$ JsonGet(ObjColumn,'owner.I*') = {"owner.id:123456} as TEXT
$ JsonGet(ObjColumn,'owner.*') = {"owner.login":"smith","owner.id":123456} as TEXT
$ JsonGet(ObjColumn,'unknown.*') = NULL
Another function, named {\f1\fs20 JsonHas()} is similar to {\f1\fs20 JsonGet()}, but will return TRUE or FALSE depending if the supplied property (specified by name or index) do exist. It may be faster to use {\f1\fs20 JsonHas()} than {\f1\fs20 JsonGet()} e.g. in a WHERE clause, when you do not want to process this property value, but only return data rows containing needed information.
$ JsonHas(ObjColumn,'owner') = true
$ JsonHas(ObjColumn,'owner.login') = true
$ JsonHas(ObjColumn,'owner.name') = false
$ JsonHas(ObjColumn,'owner.i*') = true
$ JsonHas(ObjColumn,'owner.n*') = false
Since the process will take place within the {\i SQLite3} engine itself, and since they use a @*SAX@-like fast approach (without any temporary memory allocation during its search), those JSON functions could be pretty efficient, and proudly compare to some dedicated @*NoSQL@ engines.
:     Dynamic arrays fields
:      Dynamic arrays from Delphi Code
For instance, here is how the regression @*test@s included in the framework define a {\f1\fs20 @*TSQLRecord@} class with some additional {\i @*dynamic array@s} fields:
!  TFV = packed record
!    Major, Minor, Release, Build: integer;
!    Main, Detailed: string;
!  end;
!  TFVs = array of TFV;
!  TSQLRecordPeopleArray = class(TSQLRecordPeople)
!  private
!    fInts: TIntegerDynArray;
!    fCurrency: TCurrencyDynArray;
!    fFileVersion: TFVs;
!    fUTF8: RawUTF8;
!  published
!    property UTF8: RawUTF8 read fUTF8 write fUTF8;
!!    property Ints: TIntegerDynArray index 1 read fInts write fInts;
!!    property Currency: TCurrencyDynArray index 2 read fCurrency write fCurrency;
!!    property FileVersion: TFVs index 3 read fFileVersion write fFileVersion;
!  end;
This {\f1\fs20 TSQLRecordPeopleArray} class inherits from {\f1\fs20 TSQLRecordPeople}, so it will add some new {\f1\fs20 UTF8, Ints, Currency} and {\f1\fs20 FileVersion} fields to this root class fields ({\f1\fs20 FirstName, LastName, Data, YearOfBirth, YearOfDeath}).
Some content is added to the {\f1\fs20 PeopleArray} table, with the following code:
!var V: TSQLRecordPeople;
!    VA: TSQLRecordPeopleArray;
!    FV: TFV;
!  (...)
!  V2.FillPrepare(Client,'LastName=:(''Dali''):');
!  n := 0;
!  while V2.FillOne do
!  begin
!    VA.FillFrom(V2); // fast copy some content from TSQLRecordPeople
The {\f1\fs20 FillPrepare} / {\f1\fs20 FillOne} method are used to loop through all {\f1\fs20 People} table rows with a {\f1\fs20 LastName} column value equal to 'Dali' (with a @*prepared@ statement thanks to {\f1\fs20 :( ):}), then initialize a {\f1\fs20 TSQLRecordPeopleArray} instance with those values, using a {\f1\fs20 FillFrom} method call.
!    inc(n);
!    if n and 31=0 then
!    begin
!      VA.UTF8 := '';
!!      VA.DynArray('Ints').Add(n);
!      Curr := n*0.01;
!!      VA.DynArray(2).Add(Curr);
!      FV.Major := n;
!      FV.Minor := n+2000;
!      FV.Release := n+3000;
!      FV.Build := n+4000;
!      str(n,FV.Main);
!      str(n+1000,FV.Detailed);
!!      VA.DynArray('FileVersion').Add(FV);
!    end else
!      str(n,VA.fUTF8);
The {\f1\fs20 n} variable is used to follow the {\f1\fs20 PeopleArray} number, and will most of the time set its textual converted value in the {\f1\fs20 UTF8} column, and once per 32 rows, will add one item to both {\f1\fs20 VA} and {\f1\fs20 FV} {\i @*dynamic array@} fields.
We could have used normal access to {\f1\fs20 VVA} and {\f1\fs20 FV} {\i dynamic arrays}, as such:
!     SetLength(VA.Ints,length(VA.Ints)+1);
!     VA.Ints[high(VA.Ints)] := n;
But the {\f1\fs20 DynArray} method is used instead, to allow direct access to the {\i dynamic array} via a {\f1\fs20 TDynArray} wrapper. Those two lines behave therefore the same as this code:
!      VA.DynArray('Ints').Add(n);
Note that the {\f1\fs20 DynArray} method can be used via two overloaded set of parameters: either the field name ({\f1\fs20 'Ints'}), or an {\f1\fs20 @*index@} value, as was defined in the class definition. So we could have written:
!      VA.DynArray(1).Add(n);
since the {\f1\fs20 Ints} published property has been defined as such:
!    property Ints: TIntegerDynArray
!!     index 1
!     read fInts write fInts;
Similarly, the following line will add a {\f1\fs20 @*currency@} value to the {\f1\fs20 Currency} field:
!      VA.DynArray(2).Add(Curr);
And a more complex {\f1\fs20 TFV} record is added to the {\f1\fs20 FileVersion} field {\i dynamic array} with just one line:
!      VA.DynArray('FileVersion').Add(FV);
Of course, using the {\f1\fs20 DynArray} method is a bit slower than direct {\f1\fs20 SetLength / Ints[]} use. Using {\f1\fs20 DynArray} with an index should be also a bit faster than using {\f1\fs20 DynArray} with a textual field name (like {\f1\fs20 'Ints'}), with the benefit of perhaps less keyboard errors at typing the property name. But if you need to fast add a lot of items to a {\i dynamic array}, you could use a custom {\f1\fs20 TDynArray} wrapper with an associated external {\f1\fs20 Count} value, or direct access to its content (like {\f1\fs20 SetLength + Ints[]})
Then the {\f1\fs20 FillPrepare} / {\f1\fs20 FillOne} loop ends with the following line:
!!    Check(Client.Add(VA,true)=n);
!  end;
This will add the {\f1\fs20 VA} fields content into the database, creating a new row in the {\f1\fs20 PeopleArray} table, with an {\f1\fs20 ID} following the value of the {\f1\fs20 n} variable. All {\i dynamic array} fields will be serialized as BLOB into the database table.
:21      Dynamic arrays from SQL code
In order to access the @*BLOB@ content of the dynamic arrays directly from @*SQL@ statements, some new @**SQL function@s have been defined in {\f1\fs20 TSQLDataBase}, named after their native simple types:
- {\f1\fs20 ByteDynArrayContains(BlobField,I64)};
- {\f1\fs20 WordDynArrayContains(BlobField,I64)};
- {\f1\fs20 @**IntegerDynArrayContains@(BlobField,I64)};
- {\f1\fs20 CardinalDynArrayContains(BlobField,I64)};
- {\f1\fs20 CurrencyDynArrayContains(BlobField,I64)} - in this case, {\f1\fs20 I64} is not the {\f1\fs20 @*currency@} value directly converted into an {\f1\fs20 Int64} value (i.e. not {\f1\fs20 Int64(aCurrency)}), but the binary mapping of the {\f1\fs20 currency} value, i.e. {\f1\fs20 aCurrency*10000} or {\f1\fs20 PInt64(@aCurrency)^};
- {\f1\fs20 Int64DynArrayContains(BlobField,I64)};
- {\f1\fs20 RawUTF8DynArrayContainsCase(BlobField,'Text')};
- {\f1\fs20 RawUTF8DynArrayContainsNoCase(BlobField,'Text')}.
Those functions allow direct access to the BLOB content like this:
!  for i := 1 to n shr 5 do
!  begin
!    k := i shl 5;
!!    aClient.OneFieldValues(TSQLRecordPeopleArray,'ID',
!!      FormatUTF8('IntegerDynArrayContains(Ints,?)',[],[k]),IDs);
!    Check(length(IDs)=n+1-32*i);
!    for j := 0 to high(IDs) do
!      Check(IDs[j]=k+j);
!  end;
In the above code, the WHERE clause of the {\f1\fs20 OneFieldValues} method will use the dedicated {\f1\fs20 IntegerDynArrayContains} @*SQL function@ to retrieve all records containing the specified {\f1\fs20 integer} value {\f1\fs20 k} in its {\f1\fs20 Ints} BLOB column. With such a function, all the process is performed Server-side, with no slow data transmission nor JSON/@*Base64@ @*serialization@.
For instance, using such a SQL function, you are able to store multiple {\f1\fs20 @*TSQLRecord@. ID} field values into one {\f1\fs20 TIntegerDynArray} property column, and have direct search ability inside the SQL statement. This could be a very handy way of implementing "one to many" or "many to many" relationship, without the need of a pivot table.
Those functions were implemented to be very efficient for speed. They won't create any temporary dynamic array during the search, but will access directly to the BLOB raw memory content, as returned by the {\i SQlite} engine. The {\f1\fs20 RawUTF8DynArrayContainsCase / RawUTF8DynArrayContainsNoCase} functions also will search directly inside the BLOB. With huge number of requests, this could be slower than using a {\f1\fs20 @*TSQLRecordMany@} pivot table, since the search won't use any index, and will have to read all BLOB field during the request. But, in practice, those functions behave nicely with a relative small amount of data (up to about 50,000 rows). Don't forget that BLOB column access are very optimized in {\i @*SQlite3@}.
For more complex dynamic array content handling, you'll have either to create your own @*SQL function@ using the {\f1\fs20 TSQLDataBase. RegisterSQLFunction} method and an associated {\f1\fs20 TSQLDataBaseSQLFunction} class, or via a dedicated @*Service@ or a @*stored procedure@ - see @23@ on how to implement it.
:     TPersistent/TCollection fields
For instance, here is the way regression @*test@s included in the framework define a {\f1\fs20 @*TSQLRecord@} class with some additional {\f1\fs20 @**TPersistent@}, {\f1\fs20 @**TCollection@} or {\f1\fs20 @*TRawUTF8List@} fields ({\f1\fs20 TRawUTF8List} is just a {\f1\fs20 TStringList}-like component, dedicated to handle {\f1\fs20 @*RawUTF8@} kind of {\f1\fs20 string}):
!  TSQLRecordPeopleObject = class(TSQLRecordPeople)
!  private
!    fPersistent: TCollTst;
!    fUTF8: TRawUTF8List;
!  public
!    constructor Create; override;
!    destructor Destroy; override;
!  published
!    property UTF8: TRawUTF8List read fUTF8;
!    property Persistent: TCollTst read fPersistent;
!  end;
In order to avoid any memory leak or access violation, it is mandatory to initialize then release all internal property instances in the overridden {\f1\fs20 constructor} and {\f1\fs20 destructor} of the class:
!constructor TSQLRecordPeopleObject.Create;
!begin
!  inherited;
!  fPersistent := TCollTst.Create;
!  fUTF8 := TRawUTF8List.Create;
!end;
!
!destructor TSQLRecordPeopleObject.Destroy;
!begin
!  inherited;
!  FreeAndNil(fPersistent);
!  FreeAndNil(fUTF8);
!end;
Here is how the regression @*test@s are performed:
!var VO: TSQLRecordPeopleObject;
!  (...)
!if Client.TransactionBegin(TSQLRecordPeopleObject) then
!try
!  V2.FillPrepare(Client,'LastName=?',['Morse']);
!  n := 0;
!  while V2.FillOne do
!  begin
!    VO.FillFrom(V2); // fast copy some content from TSQLRecordPeople
!    inc(n);
!    VO.Persistent.One.Color := n+100;
!    VO.Persistent.One.Length := n;
!    VO.Persistent.One.Name := Int32ToUtf8(n);
!    if n and 31=0 then
!    begin
!      VO.UTF8.Add(VO.Persistent.One.Name);
!      with VO.Persistent.Coll.Add do
!      begin
!        Color := n+1000;
!        Length := n*2;
!        Name := Int32ToUtf8(n*3);
!      end;
!    end;
!!    Check(Client.Add(VO,true)=n);
!  end;
!  Client.Commit;
!except
!  Client.RollBack; // in case of error
!end;
This will add 1000 rows to the {\f1\fs20 PeopleObject} table.
First of all, the adding is nested inside a @**transaction@ call, to speed up @*SQL@ {\f1\fs20 INSERT} statements, via {\f1\fs20 TransactionBegin} and {\f1\fs20 Commit} methods. Please note that the {\f1\fs20 TransactionBegin} method returns a {\f1\fs20 boolean} value, and should be checked in a multi-threaded or Client-Server environment (in this part of the test suit, content is accessed in the same thread, so checking the result is not mandatory, but shown here for accuracy). In the current implementation of the framework, transactions should not be nested. The typical transaction usage should be the following:
!if Client.TransactionBegin(TSQLRecordPeopleObject) then
!try
!  //.... modify the database content, raise exceptions on error
!  Client.Commit;
!except
!  Client.RollBack; // in case of error
!end;
In a @*Client-Server@ environment with multiple Clients connected at the same time, you can use the dedicated {\f1\fs20 TSQLRestClientURI.TransactionBeginRetry} method:
!if Client.TransactionBeginRetry(TSQLRecordPeopleObject,20) then
!  ...
Note that the transactions are handled according to the corresponding client @*session@: the client should make the transaction block as short as possible (e.g. using a @*batch@ command), since any write attempt by other clients will wait for the transaction to be released (with either a commit or rollback).
The fields inherited from the {\f1\fs20 @*TSQLRecord@} class are retrieved via {\f1\fs20 FillPrepare} / {\f1\fs20 FillOne} method calls, for columns with the {\f1\fs20 LastName} matching {\f1\fs20 'Morse'}. One {\f1\fs20 TPersistent} property instance values are set ({\f1\fs20 VO.Persistent.One}), then, for every 32 rows, a new item is added to the {\f1\fs20 VO.Persistent.Coll} collection.
Here is the data sent for instance to the Server, when the item with {\f1\fs20 ID=32} is added:
${"FirstName":"Samuel Finley Breese31",
$"LastName":"Morse",
$"YearOfBirth":1791,
$"YearOfDeath":1872,
$"UTF8":["32"],
$"Persistent":{"One":{"Color":132,"Length":32,"Name":"32"},"Coll":[{"Color":1032,"Length":64,"Name":"96"}]}
$}
Up to revision 1.15 of the framework, the transmitted JSON content was not a true JSON object, but sent as {\f1\fs20 @*RawUTF8@} TEXT values (i.e. every double-quote ({\f1\fs20 "}) character is escaped as {\f1\fs20 \"} - e.g. {\f1\fs20 "UTF8":"[\"32\"]"}). Starting with revision 1.16 of the framework, the transmitted data is a true JSON object, to allow better integration with an AJAX client. That is, {\f1\fs20 UTF8} field is transmitted as a valid JSON array of string, and {\f1\fs20 Persistent} as a valid JSON object with nested objects and arrays.
When all 1000 rows were added to the database file, the following loop is called once with direct connection to the DB engine, once with a remote client connection (with all available connection protocols):
!  for i := 1 to n do
!  begin
!    VO.ClearProperties;
!!    Client.Retrieve(i,VO);
!    Check(VO.ID=i);
!    Check(VO.LastName='Morse');
!    Check(VO.UTF8.Count=i shr 5);
!    for j := 0 to VO.UTF8.Count-1 do
!      Check(GetInteger(pointer(VO.UTF8[j]))=(j+1) shl 5);
!    Check(VO.Persistent.One.Length=i);
!    Check(VO.Persistent.One.Color=i+100);
!    Check(GetInteger(pointer(VO.Persistent.One.Name))=i);
!    Check(VO.Persistent.Coll.Count=i shr 5);
!    for j := 0 to VO.Persistent.Coll.Count-1 do
!     with VO.Persistent.Coll[j] do
!     begin
!       k := (j+1) shl 5;
!       Check(Color=k+1000);
!       Check(Length=k*2);
!       Check(GetInteger(pointer(Name))=k*3);
!     end;
!  end;
All the magic is made in the {\f1\fs20 Client.Retrieve(i,VO)} method. Data is retrieved from the database as TEXT values, then un-serialized from @*JSON@ arrays or objects into the internal {\f1\fs20 TRawUTF8List} and {\f1\fs20 TPersistent} instances.
When the {\f1\fs20 ID=33} row is retrieved, the following JSON content is received from the server:
${"ID":33,
$"FirstName":"Samuel Finley Breese32",
$"LastName":"Morse",
$"YearOfBirth":1791,
$"YearOfDeath":1872,
$"UTF8":"[\"32\"]",
$"Persistent":"{\"One\":{\"Color\":133,\"Length\":33,\"Name\":\"33\"},\"Coll\":[{\"Color\":1032,\"Length\":64,\"Name\":\"96\"}]}"}
In contradiction with POST content, this defines no valid nested JSON objects nor arrays, but {\f1\fs20 UTF8} and {\f1\fs20 Persistent} fields transmitted as JSON strings. This is a known limitation of the framework, due to the fact that it is much faster to retrieve directly the text from the database than process for this operation. For an AJAX application, this won't be difficult to use a temporary {\f1\fs20 string} property, and evaluate the JSON content from it, in order to replace the property with a corresponding object content. Implementation may change in the future.
:     Any TObject, including TObjectList
Not only {\f1\fs20 TPersistent, TCollection} and {\f1\fs20 TSQLRecord} types can be serialized by writing all {\f1\fs20 published} properties. The ORM core of {\i mORMot}  uses {\f1\fs20 ObjectToJSON()} and {\f1\fs20 JSONToObject()} (aka {\f1\fs20 TJSONSerializer.WriteObject}) functions to process proper JSON serialization.
You have two methods to register JSON serialization for any kind of class:
- Custom serialization via read and write callbacks - see {\f1\fs20 TJSONSerializer. @*RegisterCustomSerializer@} @52@;
- {\f1\fs20 @*TObjectList@} instances, after a proper call to {\f1\fs20 TJSONSerializer. RegisterClassForJSON} @71@.
In the database, such kind of objects will be stored as TEXT (serialized as JSON), and transmitted as regular JSON objects or arrays when working in Client-Server mode.
:     Sharding on NoSQL engines
This "Shared nothing architecture" matches perfectly with the @82@ design.
In fact, {\i mORMot}'s integration with {\i @*MongoDB@} has been optimized so that any of those high-level properties (like dynamic arrays, variants and {\i TDocVariant}, or any {\f1\fs20 class}) will be stored as BSON documents on the {\i MongoDB} server.\line If those types are able to be serialized as JSON - which is the case for simple types, {\f1\fs20 variants} and for any dynamic array / record custom types - see @53@, then the {\f1\fs20 mORMotDB.pas} unit will store this data as BSON objects or arrays on the server side, and not as BLOB or JSON text (as with SQL back-ends). You will be able to query by name any nested sub-document or sub-array, in the {\i MongoDB} collection.
As such, {\i data sharing} with {\i mORMot} will benefit of RDBMS back-end, as a reliable and proven solution, but also of the latest {\i NoSQL} technology.
:58   ORM implementation via pivot table
Data sharding just feels natural, from the @*ORM@ point of view.
But defining a pivot table is a classic and powerful use of relational database, and will unleash its power:
- When data is huge, you can query only for the needed data, without having to load the whole content (it is something similar to {\i @*lazy loading@} in ORM terminology);
- In a @*master/detail@ data model, sometimes it can be handy to access directly to the detail records, e.g. for data consolidation;
- And, last but not least, the pivot table is the natural way of storing data associated with "@*has many through@" relationship (e.g. association time or corresponding permissions).
:    Introducing TSQLRecordMany
A dedicated class, inheriting from the standard {\f1\fs20 @*TSQLRecord@} class (which is the base of all objects stored in our ORM), has been created, named {\f1\fs20 @*TSQLRecordMany@}. This table will turn the "many to many" relationship into two "one to many" relationships pointing in opposite directions. It shall contain at least two {\f1\fs20 TSQLRecord} (i.e. INTEGER) @*published properties@, named "{\f1\fs20 Source}" and "{\f1\fs20 Dest}" (name is mandatory, because the ORM will share for exact matches). The first pointing to the source record (the one with a {\f1\fs20 TSQLRecordMany} published property) and the second to the destination record.
For instance:
! TSQLDest = class(TSQLRecord);
! TSQLSource = class;
!! TSQLDestPivot = class(TSQLRecordMany)
! private
!  fSource: TSQLSource;
!  fDest: TSQLDest;
!  fTime: TDateTime;
! published
!!   property Source: TSQLSource read fSource; // map Source column
!!   property Dest: TSQLDest read fDest; // map Dest column
!   property AssociationTime: TDateTime read fTime write fTime;
! end;
! TSQLSource = class(TSQLRecordSigned)
! private
!   fDestList: TSQLDestPivot;
! published
!   property SignatureTime;
!   property Signature;
!!   property DestList: TSQLDestPivot read fDestList;
! end;
!  TSQLDest = class(TSQLRecordSigned)
!  published
!    property SignatureTime;
!    property Signature;
!  end;
When a {\f1\fs20 TSQLRecordMany} published property exists in a {\f1\fs20 TSQLRecord}, it is initialized automatically during {\f1\fs20 TSQLRecord.Create} constructor execution into a real class instance. Note that the default behavior for a {\f1\fs20 TSQLRecord} published property is to contain an {\f1\fs20 INTEGER} value which is the ID of the corresponding record - creating a "one to one" or "many to one" relationship. But {\f1\fs20 TSQLRecordMany} is a special case. So don't be confused! :)
This {\f1\fs20 TSQLRecordMany} instance is indeed available to access directly the pivot table records, via {\f1\fs20 FillMany} then {\f1\fs20 FillRow, FillOne} and {\f1\fs20 FillRewind} methods to loop through records, or {\f1\fs20 FillManyFromDest} / {\f1\fs20 DestGetJoined} for most advanced usage.
Here is how the regression @*test@s are written in the {\f1\fs20 SynSelfTests} unit:
!procedure TestMany(aClient: TSQLRestClient);
!var MS: TSQLSource;
!    MD, MD2: TSQLDest;
!    i: integer;
!    sID, dID: array[1..100] of Integer;
!    res: TIntegerDynArray;
!begin
!  MS := TSQLSource.Create;
!  MD := TSQLDest.Create;
!  try
!    MD.fSignatureTime := TimeLogNow;
!    MS.fSignatureTime := MD.fSignatureTime;
!    Check(MS.DestList<>nil);
!    Check(MS.DestList.InheritsFrom(TSQLRecordMany));
!!    aClient.TransactionBegin(TSQLSource); // faster process
This code will create two {\f1\fs20 TSQLSource / TSQLDest} instances, then will begin a @*transaction@ (for faster database engine process, since there will be multiple records added at once). Note that during {\f1\fs20 TSQLSource.Create} execution, the presence of a {\f1\fs20 TSQLRecordMany} field is detected, and the {\f1\fs20 DestList} property is filled with an instance of {\f1\fs20 TSQLDestPivot}. This {\f1\fs20 DestList} property is therefore able to be directly used via the "{\i has-many}" dedicated methods, like {\f1\fs20 ManyAdd}.
!    for i := 1 to high(dID) do
!    begin
!      MD.fSignature := FormatUTF8('% %',[aClient.ClassName,i]);
!!      dID[i] := aClient.Add(MD,true);
!      Check(dID[i]>0);
!    end;
This will just add some rows to the {\f1\fs20 Dest} table.
!    for i := 1 to high(sID) do begin
!      MS.fSignature := FormatUTF8('% %',[aClient.ClassName,i]);
!      sID[i] := aClient.Add(MS,True);
!      Check(sID[i]>0);
!      MS.DestList.AssociationTime := i;
!!      Check(MS.DestList.ManyAdd(aClient,sID[i],dID[i])); // associate both lists
!      Check(not MS.DestList.ManyAdd(aClient,sID[i],dID[i],true)); // no dup
!    end;
!    aClient.Commit;
This will create some {\f1\fs20 Source} rows, and will call the {\f1\fs20 ManyAdd} method of the auto-created {\f1\fs20 DestList} instance to associate a {\f1\fs20 Dest} item to the {\f1\fs20 Source} item. The {\f1\fs20 AssociationTime} field of the {\f1\fs20 DestList} instance is set, to implement a "{\i has many through}" relationship.
Then the @*transaction@ is committed to the database.
!    for i := 1 to high(dID) do
!    begin
!!      Check(MS.DestList.SourceGet(aClient,dID[i],res));
!      if not Check(length(res)=1) then
!        Check(res[0]=sID[i]);
!!      Check(MS.DestList.ManySelect(aClient,sID[i],dID[i]));
!      Check(MS.DestList.AssociationTime=i);
!    end;
This code will validate the association of {\f1\fs20 Source} and {\f1\fs20 Dest} tables, using the dedicated {\f1\fs20 SourceGet} method to retrieve all {\f1\fs20 Source} items {\f1\fs20 IDs} associated to the specified {\f1\fs20 Dest ID}, i.e. one item, matching the {\f1\fs20 sID[]} values. It will also check for the {\f1\fs20 AssociationTime} as set for the "{\i has many through}" relationship.
!for i := 1 to high(sID) do
!begin
!!  Check(MS.DestList.DestGet(aClient,sID[i],res));
!  if Check(length(res)=1) then
!    continue; // avoid GPF
!  Check(res[0]=dID[i]);
The {\f1\fs20 DestGet} method retrieves all {\f1\fs20 Dest} items {\f1\fs20 IDs} associated to the specified {\f1\fs20 Source ID}, i.e. one item, matching the {\f1\fs20 dID[]} values.
!!  Check(MS.DestList.FillMany(aClient,sID[i])=1);
This will fill prepare the {\f1\fs20 DestList} instance with all pivot table instances matching the specified {\f1\fs20 Source ID}. It should return only one item.
!!  Check(MS.DestList.FillOne);
!  Check(Integer(MS.DestList.Source)=sID[i]);
!  Check(Integer(MS.DestList.Dest)=dID[i]);
!  Check(MS.DestList.AssociationTime=i);
!  Check(not MS.DestList.FillOne);
Those lines will fill the first (and unique) prepared item, and check that {\f1\fs20 Source, Dest} and {\f1\fs20 AssociationTime} properties match the expected values. Then the next call to {\f1\fs20 FillOne} should fail, since only one prepared row is expected for this {\f1\fs20 Source ID}.
!!  Check(MS.DestList.DestGetJoined(aClient,'',sID[i],res));
!  if not Check(length(res)=1) then
!    Check(res[0]=dID[i]);
This will retrieve all {\f1\fs20 Dest} items {\f1\fs20 IDs} associated to the specified {\f1\fs20 Source ID}, with no additional WHERE condition.
!!  Check(MS.DestList.DestGetJoined(aClient,'Dest.SignatureTime=:(0):',sID[i],res));
!  Check(length(res)=0);
This will retrieve all {\f1\fs20 Dest} items {\f1\fs20 IDs} associated to the specified {\f1\fs20 Source ID}, with an additional always invalid WHERE condition. It should always return no item in the {\f1\fs20 res} array, since {\f1\fs20 SignatureTime} is never equal to 0.
!!  Check(MS.DestList.DestGetJoined(aClient,
!!    FormatUTF8('Dest.SignatureTime=?',[],[MD.SignatureTime]),sID[i],res));
!  if Check(length(res)=1) then
!    continue; // avoid GPF
!  Check(res[0]=dID[i]);
This will retrieve all {\f1\fs20 Dest} items {\f1\fs20 IDs} associated to the specified {\f1\fs20 Source ID}, with an additional WHERE condition, matching the expected value. It should therefore return one item.
Note the call of the global {\f1\fs20 FormatUTF8()} function to get the WHERE clause. You may have written instead:
!  Check(MS.DestList.DestGetJoined(aClient,
!    'Dest.SignatureTime=:('+Int64ToUTF8(MD.SignatureTime)+'):',sID[i],res));
But in this case, using manual inlined {\f1\fs20 :(..):} values is less convenient than the '?' calling convention, especially for string ({\f1\fs20 @*RawUTF8@}) values.
!!  MD2 := MS.DestList.DestGetJoined(aClient,
!!    FormatUTF8('Dest.SignatureTime=?',[],[MD.SignatureTime]),sID[i]) as TSQLADest;
!  if Check(MD2<>nil) then
!    continue;
!  try
!!    Check(MD2.FillOne);
!    Check(MD2.ID=dID[i]);
!    Check(MD2.Signature=FormatUTF8('% %',[aClient.ClassName,i]));
!  finally
!    MD2.Free;
!  end;
!end;
This overloaded {\f1\fs20 DestGetJoined} method will return into {\f1\fs20 MD2} a {\f1\fs20 TSQLDest} instance, prepared with all the {\f1\fs20 Dest} record content associated to the specified {\f1\fs20 Source ID} , with an additional WHERE condition, matching the expected value. Then {\f1\fs20 FillOne} will retrieve the first (and unique) matching {\f1\fs20 Dest} record, and checks for its values.
!!    aClient.TransactionBegin(TSQLADests); // faster process
!    for i := 1 to high(sID) shr 2 do
!!      Check(MS.DestList.ManyDelete(aClient,sID[i*4],dID[i*4]));
!!    aClient.Commit;
!    for i := 1 to high(sID) do
!      if i and 3<>0 then
!      begin
!!        Check(MS.DestList.ManySelect(aClient,sID[i],dID[i]));
!        Check(MS.DestList.AssociationTime=i);
!      end else
!        Check(not MS.DestList.ManySelect(aClient,sID[i],dID[i]));
This code will delete one association per four, and ensure that {\f1\fs20 ManySelect} will retrieve only expected associations.
!  finally
!    MD.Free;
!    MS.Free;
!  end;
This will release associated memory, and also the instance of {\f1\fs20 TSQLDestPivot} created in the {\f1\fs20 DestList} property.
:    Automatic JOIN query
All those methods ({\f1\fs20 ManySelect, DestGetJoined...}) are used to retrieve the relations between tables from the pivot table point of view. This saves bandwidth, and can be used in most simple cases, but it is not the only way to perform requests on many-to-many relationships. And you may have several {\f1\fs20 @*TSQLRecordMany@} instances in the same main record - in this case, those methods won't help you.
It is very common, in the SQL world, to create a @**JOIN@ed request at the main "{\i Source}" table level, and combine records from two or more tables in a database. It creates a set that can be saved as a table or used as is. A JOIN is a means for combining fields from two or more tables by using values common to each. Writing such JOINed statements is not so easy by hand, especially because you'll have to work with several tables, and have to specify the exact fields to be retrieved; if you have several pivot tables, it may start to be a nightmare. Let's see how our @*ORM@ will handle it.
A dedicated {\f1\fs20 FillPrepareMany} method has been added to the {\f1\fs20 @*TSQLRecord@} class, in conjunction with a new {\f1\fs20 constructor} named {\f1\fs20 CreateAndFillPrepareMany}. This particular method will:
- Instantiate all {\f1\fs20 Dest} properties of each {\f1\fs20 TSQLRecordMany} instances - so that the JOINed request will be able to populate directly those values;
- Create the appropriate {\f1\fs20 SELECT} statement, with an optional WHERE clause.
Here is the test included in our regression suite, working with the same database:
!Check(MS.FillPrepareMany(aClient,
!  'DestList.Dest.SignatureTime<>% and id>=? and DestList.AssociationTime<>0 '+
!  'and SignatureTime=DestList.Dest.SignatureTime '+
!  'and DestList.Dest.Signature<>"DestList.AssociationTime"',[0],[sID[1]]));
Of course, the only useful parameter here is {\f1\fs20 id>=?} which is used to retrieve the just added relationships in the pivot table. All other conditions will always be true, but it will help testing the generated SQL.
Our {\i mORMot} will generate the following SQL statement:
$select A.ID AID,A.SignatureTime A00,A.Signature A01,
$  B.ID BID,B.AssociationTime B02,
$  C.ID CID,C.SignatureTime C00,C.Signature C01
$from ASource A,ADests B,ADest C
$where B.Source=A.ID and B.Dest=C.ID
$  and (C.SignatureTime<>0 and A.id>=:(1): and B.AssociationTime<>0
$  and A.SignatureTime=C.SignatureTime and C.Signature<>"DestList.AssociationTime")
You can notice the following:
- All declared {\f1\fs20 TSQLRecordMany} instances (renamed {\f1\fs20 B} in our case) are included in the statement, with all corresponding {\f1\fs20 Dest} instances (renamed as {\f1\fs20 C});
- Fields are aliased with short unique identifiers ({\f1\fs20 AID, A01, BID, B02...}), for all {\i simple} properties of every classes;
- The JOIN clause is created ({\f1\fs20 B.Source=A.ID and B.Dest=C.ID});
- Our manual WHERE clause has been translated into proper SQL, including the table internal aliases ({\f1\fs20 A,B,C}) - in fact, {\f1\fs20 DestList.Dest} has been replaced by {\f1\fs20 C}, the main {\f1\fs20 ID} property has been declared properly as {\f1\fs20 A.ID}, and the {\f1\fs20 "DestList.AssociationTime"} text remained untouched, because it was bounded with quotes.
That is, our @*ORM@ did make all the dirty work for you! You can use {\i Delphi}-level conditions in your query, and the engine will transparently convert them into a valid SQL statement. Benefit of this will become clear in case of multiple pivot tables, which are likely to occur in real-world applications.
After the statement has been prepared, you can use the standard {\f1\fs20 FillOne} method to loop through all returned rows of data, and access to the JOINed columns within the {\i Delphi} objects instances:
!  Check(MS.FillTable.RowCount=length(sID));
!  for i := 1 to high(sID) do begin
!!   MS.FillOne;
!    Check(MS.fID=sID[i]);
!    Check(MS.SignatureTime=MD.fSignatureTime);
!    Check(MS.DestList.AssociationTime=i);
!    Check(MS.DestList.Dest.fID=dID[i]);
!    Check(MS.DestList.Dest.SignatureTime=MD.fSignatureTime);
!    Check(MS.DestList.Dest.Signature=FormatUTF8('% %',[aClient.ClassName,i]));
!  end;
!!  MS.FillClose;
Note that in our case, an explicit call to {\f1\fs20 FillClose} has been added in order to release all {\f1\fs20 Dest} instances created in {\f1\fs20 FillPrepareMany}. This call is not mandatory if you call {\f1\fs20 MS.Free} directly, but it is required if the same {\f1\fs20 MS} instance is about to use some regular many-to-many methods, like {\f1\fs20 MS.DestList.ManySelect()} - it will prevent any GPF exception to occur with code expecting the {\f1\fs20 Dest} property not to be an instance, but a {\f1\fs20 pointer(DestID)} value.
\page
:110 ORM Data Model
:  Creating an ORM Model
The {\f1\fs20 @**TSQLModel@} class centralizes all {\f1\fs20 @*TSQLRecord@} inherited classes used by an application, both database-related and business-logic related.
In order to follow the @*MVC@ pattern, the {\f1\fs20 TSQLModel} instance is to be used when you have to deal at table level. For instance, do not try to use low-level {\f1\fs20 TSQLDataBase.GetTableNames} or {\f1\fs20 TSQLDataBase.GetFieldNames} methods in your code. In fact, the tables declared in the Model may not be available in the {\i @*SQLite3@} database schema, but may have been defined as {\f1\fs20 @*TSQLRestStorageInMemory@} instance via the {\f1\fs20 TSQLRestServer.StaticDataCreate} method, or being external tables - see @27@. You could even have a {\f1\fs20 mORMot} server running without any {\i @*SQLite3@} engine at all, but pure in-memory tables!
Each {\f1\fs20 TSQLModel} instance is in fact associated with a {\f1\fs20 TSQLRest} instance. An {\f1\fs20 Owner} property gives access to the current running client or server {\f1\fs20 @*TSQLRest@} instance associated with this model.
By design, models are used on both Client and Server sides. It is therefore a good practice to use a common unit to define all {\f1\fs20 TSQLRecord} types, and have a common function to create the related {\f1\fs20 TSQLModel} class.
For instance, here is the corresponding function as defined in the first samples available in the source code repository (unit {\f1\fs20 SampleData.pas}):
!function CreateSampleModel: TSQLModel;
!begin
!  result := TSQLModel.Create([TSQLSampleRecord]);
!end;
For a more complex model including link to User Interface, see @64@.
:163  Several Models
In practice, a same {\f1\fs20 TSQLRecord} can be used in several models: this is typically the case for {\f1\fs20 TSQLAuthUser} tables, or if client and server instances are running in the same process. So, for accessing the model properties, you have two structures available:
|%37%68
|\b Class|Description\b0
|{\f1\fs20 @**TSQLModelRecordProperties@}|Model-specific ORM parameters, like dedicated SQL auto-generation and external DB settings.\line Access these instances from {\f1\fs20 TSQLModel.TableProps[]} array
|{\f1\fs20 @**TSQLRecordProperties@}|Low-level table properties, as retrieved from @5@ by the ORM kernel of {\i mORMot}.\line Access these instances from {\f1\fs20 TSQLModel.TableProps[].Props}
|%
So you may use code like this:
!var i: integer;
!    ModelProps: TSQLModelRecordProperties;
!    Props: TSQLRecordProperties;
!begin
!  ...
!  for i := 0 to high(Model.TableProps) do begin
!    ModelProps := Model.TableProps[i];
!    // now you can access ModelProps.ExternalDB.TableName ...
!    Props := ModelProps.Props;
!    // now you can use Props.SQLTableName or Props.Fields[]
!  end;
!end;
:56  Filtering and Validating
According to the n-Tier architecture - see @7@ - data @**filtering@ and @**validation@ should be implemented in the business logic, not in the User Interface.
If you were used to develop RAD database application using {\i Delphi}, you may have to change a bit your habits here. Data filtering and validation should be implemented not in the User Interface, but in pure {\i Delphi} code.
In order to make this easy, a dedicated set of classes are available in the {\f1\fs20 SynCommons.pas} unit, and allow to define both filtering (transformation) and validation. They all will be children of any of those both classes:
\graph HierTSynFilter Filtering and Validation classes hierarchy
\TSynValidate\TSynFilterOrValidate
\TSynFilter\TSynFilterOrValidate
\
{\f1\fs20 @*TSQLRecord@} field content {\i filtering} is handled in the {\f1\fs20 TSQLRecord. Filter} virtual method, or via some {\f1\fs20 TSQLFilter} classes. They will {\i transform} the object fields following some rules, e.g. forcing uppercase/lowercase, or trimming text spaces.
{\f1\fs20 TSQLRecord} field content {\i validation} is handled in the {\f1\fs20 TSQLRecord. Validate} virtual method, or via some {\f1\fs20 TSQLValidate} classes. Here the object fields will be checked against a set of rules, and report any invalid content.
Some "standard" classes are already defined in the {\f1\fs20 SynCommons.pas} and {\f1\fs20 mORMot.pas} units, covering most common usage:
\graph HierTSynFilters Default filters and Validation classes hierarchy
\TSynValidatePassWord\TSynValidateText
\TSynValidateNonVoidText\TSynValidate
\TSynValidateText\TSynValidate
\TSynValidateTableUniqueField\TSynValidateTable
\TSynValidateTable\TSynValidate
\TSynValidateUniqueField\TSynValidateRest
\TSynValidateRest\TSynValidate
\TSynValidatePatternI\TSynValidatePattern
\TSynValidatePattern\TSynValidate
\TSynValidateIPAddress\TSynValidate
\TSynValidateEmail\TSynValidate
\TSynValidate\TSynFilterOrValidate
\TSynFilterUpperCaseU\TSynFilter
\TSynFilterUpperCase\TSynFilter
\TSynFilterTrim\TSynFilter
\TSynFilterLowerCaseU\TSynFilter
\TSynFilterLowerCase\TSynFilter
\TSynFilterTruncate\TSynFilter
\TSynFilter\TSynFilterOrValidate
rankdir=LR;
\
You have powerful validation classes for IP Address, Email (with TLD+domain name), simple {\i regex} pattern, textual validation, strong password validation...
Note that some database-related validation is existing, like {\f1\fs20 TSynValidateUniqueField} which inherits from {\f1\fs20 TSynValidateRest}.
Of course, the {\f1\fs20 mORMotUIEdit} unit handles {\f1\fs20 @*TSQLRecord@} automated filtering (using {\f1\fs20 TSQLFilter} classes) and validation (via the {\f1\fs20 TSQLValidate} classes).
The field validation process is run in {\f1\fs20 TSQLRecord. Validate} and not in {\f1\fs20 mORMotUIEdit} itself (to have a better multi-tier architecture).
To initialize it, you can add some filters/validators to your {\f1\fs20 @*TSQLModel@} creation function:
!function CreateFileModel(Owner: TSQLRest): TSQLModel;
!begin
!  result := TSQLModel.Create(Owner,
!    @FileTabs,length(FileTabs),sizeof(FileTabs[0]),[],
!    TypeInfo(TFileAction),TypeInfo(TFileEvent),'synfile');
!  TSQLFile.AddFilterOrValidate('Name',TSQLFilterLowerCase);
!  TSQLUser.AddFilterOrValidate('Email',TSQLValidateEmail);
!end;
As an alternative, you can override the following method:
!  TSQLRecordAuthInfo = class(TSQLRecord)
!  protected
!!    class procedure InternalDefineModel(Props: TSQLRecordProperties); override;
!  ...
!
!class procedure TSQLRecordAuthInfo.InternalDefineModel(
!  Props: TSQLRecordProperties);
!begin
!  AddFilterNotVoidText(['Logon','HashedPassword']);
!end;
It does make sense to define this behavior within the {\f1\fs20 TSQLRecord} definition, so that it will be shared by all models.
If you want to perform some text field length validation or filter at ORM level, you may use {\f1\fs20 TSQLRecordProperties}'s {\f1\fs20 SetMaxLengthValidatorForTextFields()} or {\f1\fs20 SetMaxLengthFilterForTextFields()} method, or at model level:
!function CreateModel: TSQLModel;
!begin
!  result := TSQLModel.Create([TSQLMyRecord1,TSQLMyRecord2]);
!  result.SetMaxLengthValidatorForAllTextFields(true); // "index n" is in UTF-8 bytes
!end;
In order to perform the filtering (transformation) of some content, you'll have to call the {\f1\fs20 aRecord.Filter()} method, and {\f1\fs20 aRecord.Validate()} to test for valid content.
For instance, this is how {\f1\fs20 mORMotUIEdit.pas} unit filters and validates the user interface input:
!procedure TRecordEditForm.BtnSaveClick(Sender: TObject);
! (...)
!  // perform all registered filtering
!!  Rec.Filter(ModifiedFields);
!  // perform content validation
!  FieldIndex := -1;
!!  ErrMsg := Rec.Validate(Client,ModifiedFields,@FieldIndex);
!  if ErrMsg<>'' then begin
!    // invalid field content -> show message, focus component and abort saving
!    if cardinal(FieldIndex)<cardinal(length(fFieldComponents)) then begin
!      C := fFieldComponents[FieldIndex];
!      C.SetFocus;
!      Application.ProcessMessages;
!      ShowMessage(ErrMsg,format(sInvalidFieldN,[fFieldCaption[FieldIndex]]),true);
!    end else
!      ShowMessage(ErrMsg,format(sInvalidFieldN,['?']),true);
!  end else
!    // close window on success
!    ModalResult := mrOk;
!end;
It is up to your code to filter and validate the record content. By default, the {\i mORMot} @*CRUD@ operations won't call the registered filters or validators.
:38 ORM Cache
Here is the definition of "cache", as stated by {\i Wikipedia}:
{\i In computer engineering, a @**cache@ is a component that transparently stores data so that future requests for that data can be served faster. The data that is stored within a cache might be values that have been computed earlier or duplicates of original values that are stored elsewhere. If requested data is contained in the cache (cache hit), this request can be served by simply reading the cache, which is comparatively faster. Otherwise (cache miss), the data has to be recomputed or fetched from its original storage location, which is comparatively slower. Hence, the greater the number of requests that can be served from the cache, the faster the overall system performance becomes.}
{\i To be cost efficient and to enable an efficient use of data, caches are relatively small. Nevertheless, caches have proven themselves in many areas of computing because access patterns in typical computer applications have locality of reference. References exhibit temporal locality if data is requested again that has been recently requested already. References exhibit spatial locality if data is requested that is physically stored close to data that has been requested already.}
In our @*ORM@ framework, since performance was one of our goals since the beginning, cache has been implemented at four levels:
- @*Statement@ cache for implementing @*SQL@ @*prepared@ statements, and parameters bound on the fly - see @36@ and @14@ - note that this cache is available not only for the {\i @*SQlite3@} database engine, but also for any external engine - see @27@;
- Global @*JSON@ result cache at the database level, which is flushed globally on any {\f1\fs20 INSERT / UPDATE} - see @37@;
- Tuned record cache at the @*CRUD@/@*REST@ful level for specified tables or records on the {\i server} side - see @39@;
- Tuned record cache at the CRUD/RESTful level for specified tables or records on the {\i client} side - see @39@.
Thanks to those specific caching abilities, our framework is able to minimize the number of client-server requests, therefore spare bandwidth and network access, and scales well in a concurrent rich client access architecture. In such perspective, a @*Client-Server@ ORM does make sense, and is of huge benefit in comparison to a basic ORM used only for data persistence and automated SQL generation.
\page
: Calculated fields
It is often useful to handle some calculated fields. That is, having some field values computed when you set another field value. For instance, if you set an error code from an enumeration (stored in an INTEGER field), you may want the corresponding text (to be stored on a TEXT field). Or you may want a total amount to be computed automatically from some detailed records.
This should not be done on the Server side. In fact, the framework expects the transmitted JSON transmitted from client to be set directly to the database layer, as stated by this code from the {\f1\fs20 mORMotSQLite3} unit:
!function TSQLRestServerDB.EngineUpdate(Table: TSQLRecordClass; ID: TID;
!  const SentData: RawUTF8): boolean;
!begin
!  if (self=nil) or (Table=nil) or (ID<=0) then
!    result := false else begin
!    // this SQL statement use :(inlined params): for all values
!    result := ExecuteFmt('UPDATE % SET % WHERE RowID=:(%):;',
!!      [Table.RecordProps.SQLTableName,GetJSONObjectAsSQL(SentData,true,true),ID]);
!    if Assigned(OnUpdateEvent) then
!       OnUpdateEvent(self,seUpdate,Table,ID);
!  end;
!end;
The direct conversion from the received JSON content into the SQL {\f1\fs20 UPDATE} statement values is performed very quickly via the {\f1\fs20 GetJSONObjectAsSQL} procedure. It won't use any intermediary {\f1\fs20 @*TSQLRecord@}, so there will be no server-side field calculation possible.
Record-level calculated fields should be done on the Client side, using some setters.
There are at least three ways of updating field values before sending to the server:
- Either by using some dedicated setters method for {\f1\fs20 TSQLRecord} properties;
- Or by overriding the {\f1\fs20 ComputeFieldsBeforeWrite} virtual method of {\f1\fs20 TSQLRecord}.
- If the computed fields need a more complex implementation (e.g. if some properties of another record should be modified), a dedicated @*REST@ful @*service@ should be implemented - see @11@.
:  Setter for TSQLRecord
For instance, here we define a new table named {\i INVOICE}, with only two fields. A @*dynamic array@ containing the invoice details, then a field with the total amount. The dynamic array property will be stored as BLOB into the database, and no additional @*Master/Detail@ table will be necessary.
!type
!  TInvoiceRec = record
!    Ident: RawUTF8;
!    Amount: currency;
!  end;
!  TInvoiceRecs = array of TInvoiceRec;
!  TSQLInvoice = class(TSQLRecord)
!  protected
!    fDetails: TInvoiceRecs;
!    fTotal: Currency;
!    procedure SetDetails(const Value: TInvoiceRecs);
!  published
!    property Details: TInvoiceRecs read fDetails write SetDetails;
!    property Total: Currency read fTotal;
!  end;
Note that the {\f1\fs20 Total} property does not have any {\i setter} (aka {\f1\fs20 write} statement). So it will be read-only, from the ORM point of view. In fact, the following protected method will compute the {\f1\fs20 Total} property content from the {\f1\fs20 Details} property values, when they will be modified:
!procedure TSQLInvoice.SetDetails(const Value: TInvoiceRecs);
!var i: integer;
!begin
!  fDetails := Value;
!  fTotal := 0;
!  for i := 0 to high(Value) do
!    fTotal := fTotal+Value[i].Amount;
!end;
When the object content will be sent to the Server, the {\f1\fs20 Total} value of the JSON content sent will contain the expected value.
Note that with this implementation, the {\f1\fs20 SetDetails} must be called explicitly. That is, you should {\i not only} modify directly the {\f1\fs20 Details[]} array content, but either use a temporary array during edition then assign its value to {\f1\fs20 Invoice.Details}, or force the update with a line of code like:
! Invoice.Details := Invoice.Details; // force Total calculation
:  TSQLRecord.ComputeFieldsBeforeWrite
Even if a {\f1\fs20 @*TSQLRecord@} instance should not normally have access to the {\f1\fs20 TSQLRest} level, according to @*OOP@ principles, the following virtual method have been defined:
!  TSQLRecord = class(TObject)
!  public
!    procedure ComputeFieldsBeforeWrite(aRest: TSQLRest; aOccasion: TSQLEvent); virtual;
!  (...)
It will be called automatically on the Client side, just before a {\f1\fs20 TSQLRecord} content will be sent to the remote server, before adding or update.
In fact, the {\f1\fs20 TSQLRestClientURI.Add / Update / BatchAdd / BatchUpdate} methods will call this method before calling {\f1\fs20 TSQLRecord.GetJSONValues} and send the JSON content to the server.
On the Server-side, in case of some business logic involving the ORM, the {\f1\fs20 TSQLRestServer.Add / Update} methods will also call {\f1\fs20 ComputeFieldsBeforeWrite}.
By default, this method will compute the {\f1\fs20 @*TModTime@ / sftModTime} and {\f1\fs20 @*TCreateTime@ / sftCreateTime}  properties value from the current @*server time stamp@, as such:
!procedure TSQLRecord.ComputeFieldsBeforeWrite(aRest: TSQLRest; aOccasion: TSQLEvent);
!var F: integer;
!begin
!  if (self<>nil) and (aRest<>nil) then
!    with RecordProps do begin
!      if HasModTimeFields then
!        for F := 0 to high(FieldType) do
!        if FieldType[f]=sftModTime then
!!          SetInt64Prop(Self,Fields[F],aRest.ServerTimestamp);
!      if HasCreateTimeField and (aOccasion=seAdd) then
!        for F := 0 to high(FieldType) do
!        if FieldType[f]=sftCreateTime then
!!          SetInt64Prop(Self,Fields[F],aRest.ServerTimestamp);
!    end;
!end;
You may override this method for you own purpose, saved the fact that you call this inherited implementation to properly handle {\f1\fs20 TModTime} and {\f1\fs20 TCreateTime} @*published properties@.
:85 Audit Trail for change tracking
Since most @*CRUD@ operations are centered within the scope of our {\i mORMot} server, we implemented in the @*ORM@ an integrated mean of tracking changes (aka @**Audit Trail@) of any {\f1\fs20 TSQLRecord}.
Keeping a track of the history of business objects is one very common need for software modeling, and a must-have for any accurate data @*model@ing, like @54@. By default, as expected by the @*OOP@ model, any change to an object will forget any previous state of this object. But thanks to {\i mORMot}'s exclusive change-tracking feature, you can persist the history of your objects.
:  Enabling audit-trail
By default, change-tracking feature will be disabled, saving performance and disk use.\line But you can enable change tracking for any class, by calling the following method, on server side:
! aServer.TrackChanges([TSQLInvoice]);
This single line will let {\f1\fs20 aServer: TSQLRestServer} monitor all CRUD operations, and store all changes of the {\f1\fs20 TSQLInvoice} table within a {\f1\fs20 TSQLRecordHistory} table.
Since all content changes will be stored in this single table by default (note that the {\f1\fs20 TrackChanges()} method accepts an {\i array of classes} as parameters, and can be called several times), it may be handy to define several tables for history storage. Later on, an external database engine may be defined to store history, e.g. on cheap hardware (and big hard drives), whereas your main database may be powered by high-end hardware (and smaller SSDs) - see @27@.\line To do so, you define your custom class for history storage, then supply it as parameter:
!type
!  TSQLRecordSecondaryHistory = class(TSQLRecordHistory);
! (...)
! aServer.TrackChanges([TSQLInvoice],TSQLRecordSecondaryHistory);
Then, all history will be stored in this {\f1\fs20 TSQLRecordSecondaryHistory} class (in its own table named {\f1\fs20 SecondaryHistory}), and not the default {\f1\fs20 TSQLRecordHistory} class (in its {\f1\fs20 History} table).
:  A true Time Machine for your objects
Once the object changes are tracked, you can later on browse the history of the object, by using the {\f1\fs20 TSQLRecordHistory.CreateHistory()}, then {\f1\fs20 HistoryGetLast}, {\f1\fs20 HistoryCount}, and {\f1\fs20 HistoryGet()} methods:
!var aHist: TSQLRecordHistory;
!    aInvoice: TSQLInvoice;
!    aEvent: TSQLHistoryEvent; // will be either heAdd, heUpdate or heDelete
!    aTimestamp: TModTime;
!(...)
!aInvoice := TSQLInvoice.Create;
!aHist := TSQLRecordHistory.CreateHistory(aClient,TSQLInvoice,400);
!try
!  writeln('Number of items in the record history: ',aHist.HistoryCount);
!  for i := 0 to aHist.HistoryCount-1 do begin
!    aHist.HistoryGet(i,aEvent,aTimestamp,aInvoice);
!    writeln;
!    writeln('Event: ',GetEnumName(TypeInfo(TSQLHistoryEvent),ord(aEvent))^);
!    writeln('Timestamp: ',TTimeLogBits(aTimestamp).ToText);
!    writeln('Identifier: ',aInvoice.Number);
!    writeln('Value: ',aInvoice.GetJSONValues(true,true,soSelect));
!  end;
!finally
!  aHist.Free;
!  aInvoice.Free;
!end;
Note that you have several overloaded versions of {\f1\fs20 TSQLRecordHistory.HistoryGet()} to retrieve the record values.
As a result, our ORM is also transformed into a true {\i time machine}, for the objects which need it.
This feature will be available on both client and server sides, via the {\f1\fs20 TSQLRecordHistory} table.
:  Automatic history packing
This {\f1\fs20 TSQLRecordHistory} class will in fact create a {\f1\fs20 History} table in the main database, defined as such:
\graph DBHistory History Record Layout
rankdir=LR;
node [shape=Mrecord];
struct1 [label="ID : TID|Event : TSQLHistoryEvent|History : TSQLRawBlob|ModifiedRecord : PtrInt|SentDataJSON : RawUTF8|Timestamp : TModTime"];
\
In short, any modification via the ORM will be stored in the {\f1\fs20 TSQLRecordHistory} table, as a JSON object of the changed fields, in {\f1\fs20 TSQLRecordHistory.SentDataJSON}.
By design, direct SQL changes are not handled. If you run some SQL statements like {\f1\fs20 DELETE FROM ...} or {\f1\fs20 UPDATE ... SET ...} within your application or from any external program, then the History table won't be updated.\line In fact, the ORM does not set any DB trigger to track low-level changes: it will slow down the process, and void the {\i persistence agnosticism} paradigm we want to follow, e.g. allowing to use a @*NoSQL@ database like @*MongoDB@.
When the history grows, the JSON content may become huge, and fill the disk space with a lot of duplicated content. In order to save disk space, when a record reaches a define number of JSON data rows, all this JSON content is gathered and compressed into a BLOB, in {\f1\fs20 TSQLRecordHistory.History}.\line You can force this packing process by calling {\f1\fs20 TSQLRestServer.TrackChangesFlush()} manually in your code. Calling this method will also have a welcome side effect: it will read the actual content of the record from the database, then add a fake {\f1\fs20 heUpdate} event in the history if the field values do not match the one computed from tracked changes, to ensure that the audit trail will be correct. As a consequence, history will become always synchronized with the actual data persisted in the database, even if external SQL did by-pass the CRUD methods of the ORM, via unsafe {\f1\fs20 DELETE FROM ...} or {\f1\fs20 UPDATE ... SET ...} statements.
You can tune how packing is defined for a given {\f1\fs20 TSQLRecord} table, by using some optional parameters to the registering method:
!procedure TrackChanges(const aTable: array of TSQLRecordClass;
!  aTableHistory: TSQLRecordHistoryClass=nil; aMaxHistoryRowBeforeBlob: integer=1000;
!  aMaxHistoryRowPerRecord: integer=10; aMaxUncompressedBlobSize: integer=64*1024); virtual;
Take a look at the documentation of this method (or the comments in its declaration code) for further information.\line Default options will let {\f1\fs20 TSQLRestServer.TrackChangesFlush()} be called after 1000 individual {\f1\fs20 TSQLRecordHistory.SentDataJSON} rows are stored, then will compress them into a BLOB once 10 JSON rows are available for a given record, ensuring that the uncompressed BLOB size for a single record won't use more than 64 KB of memory (but probably much less in the database, since it is stored with very high compression rate).
:147 Master/slave replication
As stated during @26@, the ORM is able to maintain a revision number for any {\f1\fs20 TSQLRecord} table, so that it the table may be easily synchronized remotely by another {\f1\fs20 TSQLRestServer} instance.\line If you define a {\f1\fs20 @**TRecordVersion@} published property, the ORM core will fill this field just before any write with a monotonically increasing revision number, and will take care of any deletion, so that those modifications may be replayed later on any other database.
This synchronization will work as a strict @**master/slave@ @**replication@ scheme, as a one-way on demand refresh of a replicated table. Each write operation on the master database on a given table may be easily reflected on one or several slave databases, with almost no speed nor storage size penalty.
In addition to this {\i on demand} synchronization, a real-time notification mechanism, using {\i @*WebSockets@} communication - see @150@ - may be defined.
:  Enable synchronization
In order to enable this @*replication@ mechanism, you should define a {\f1\fs20 @TRecordVersion@} published property in the {\f1\fs20 TSQLRecord} class type definition:
!  TSQLRecordPeopleVersioned = class(TSQLRecordPeople)
!  protected
!    fFirstName: RawUTF8;
!    fLastName: RawUTF8;
!    fVersion: TRecordVersion;
!  published
!    property FirstName: RawUTF8 read fFirstName write fFirstName;
!    property LastName: RawUTF8 read fLastName write fLastName;
!!    property Version: TRecordVersion read fVersion write fVersion;
!  end;
Only a single {\f1\fs20 TRecordVersion} field is allowed per {\f1\fs20 TSQLRecord} class - it will not mean anything to manage more than one field of this type.
Note that this field will be somewhat "hidden" to most ORM process: a regular {\f1\fs20 TSQLRest.Retrieve} won't fill this {\f1\fs20 Version} property, since it is an internal implementation detail. If you want to lookup its value, you will have to explicitly state its field name at retrieval. Any {\f1\fs20 TRecordVersion} is indeed considered as a "non simple field", just like BLOB fields, so will need explicit retrieval of its value.
In practice, any {\f1\fs20 TSQLRest.Add} and {\f1\fs20 TSQLRest.Update} on this {\f1\fs20 TSQLRecordPeopleVersioned} class will increase this {\f1\fs20 Version} revision number field, and a {\f1\fs20 TSQLRest.Delete} will populate an external {\f1\fs20 @*TSQLRecordTableDelete@} table with the {\f1\fs20 ID} of the deleted record, associated with a {\f1\fs20 TRecordVersion} revision.
As consequences:
- The monotonic {\f1\fs20 TRecordVersion} number is shared at {\f1\fs20 TSQLRestServer} level, among all tables containing a {\f1\fs20 TRecordVersion} published field;
- The {\f1\fs20 TSQLRecordTableDelete} table should be part of the {\f1\fs20 TSQLModel}, in conjunction with {\f1\fs20 TSQLRecordPeopleVersioned};
- If the {\f1\fs20 TSQLRecordTableDelete} table is not part of the {\f1\fs20 TSQLModel}, the {\f1\fs20 TSQLRestServer} will add it - but you should better make it explicitly appearing in the data model;
- A single {\f1\fs20 TSQLRecordTableDelete} table will maintain the list of all deleted data rows, of all tables containing a {\f1\fs20 TRecordVersion} published field;
- The {\f1\fs20 TSQLRecordPeopleVersioned} table appearance order in the {\f1\fs20 TSQLModel} will matter, since {\f1\fs20 TSQLRecordTableDelete.ID} will use this table index order in the database model to identify the table type of the deleted row - in a similar way to @148@.
All the synchronization preparation will be taken care by the ORM kernel on its own, during any write operation. There is nothing particular to maintain or setup, in addition to this {\f1\fs20 TRecordVersion} field definition, and the global {\f1\fs20 TSQLRecordTableDelete} table.
:  From master to slave
To replicate this {\f1\fs20 TSQLRecordPeopleVersioned} table from another {\f1\fs20 TSQLRestServer} instance, just call the following method:
! aServer.RecordVersionSynchronizeSlave(TSQLRecordPeopleVersioned,aClient);
This single line will request a remote server via a {\f1\fs20 Client: TSQLRestClientURI} connection (which may be over @*HTTP@) for any pending modifications since its last call, then will fill the local {\f1\fs20 aServer: TSQLRestServer} database so that the local {\f1\fs20 TSQLRecordPeopleVersioned} table will contain the very same content as the remote master {\f1\fs20 TSQLRestServer}.
You can safely call {\f1\fs20 TSQLRestServer.RecordVersionSynchronizeSlave} from several clients, to replicate the master data in several databases.
Using a {\f1\fs20 TTimer} may increase responsiveness of a client application, and allow refresh of displayed data, with limited resources (e.g. with a 500 ms period, on a given screen).
Only the modified data will be transmitted over the wire, as two REST/JSON queries (one for the insertions/updates, another for the deletions), and all the local write process will use optimized BATCH writing - see @28@. This means that the synchronization process will try to use as minimal bandwidth and resources as possible, on both sides.
In practice, you may define the {\i Master} side as such:
!  MasterServer := TSQLRestServerDB.Create(MasterModel,'master.db3');
!  HttpMasterServer := TSQLHttpServer.Create('8888',[MasterServer],'+',useBidirSocket);
On the {\i Slave} side, the HTTP client will access the {\i Master} database as usual:
!  MasterClient := TSQLHttpClientWebsockets.Create('127.0.0.1',HTTP_DEFAULTPORT,MasterModel);
Of course, the model should match for both {\f1\fs20 MasterServer} and {\f1\fs20 MasterClient} instances. This is why we used the same {\f1\fs20 MasterModel} variable name (probably defined in a shared unit).
Assuming that the {\i slave} database has been defined as such:
!  SlaveServer := TSQLRestServerDB.Create(SlaveModel,'slave.db3');
Then you can run replication from the {\i slave} side with a single line, for a given table:
!  SlaveServer.RecordVersionSynchronizeSlave(TSQLRecordPeopleVersioned,MasterClient);
This command will process the replication as such:
\graph ORMReplication01 ORM Replication Classes via REST
subgraph cluster_0 {
label="Master";
\MasterServer¤(MasterModel)\master.db3
}
subgraph cluster_2 {
label="       Slave";
\MasterServer¤(MasterModel)\MasterClient¤(MasterModel)\HTTP
\MasterClient¤(MasterModel)\MasterServer¤(MasterModel)
\MasterClient¤(MasterModel)\SlaveServer¤(SlaveModel)\On Demand¤Replication
\SlaveServer¤(SlaveModel)\MasterClient¤(MasterModel)
\SlaveServer¤(SlaveModel)\slave.db3
}
\
Of course, the slaves should be considered as read-only, otherwise the version numbers may conflict, and the whole synchronization may become a failure.
But you can safely replicate servers in cascade, if needed: the version numbers will be propagated from masters to slaves, and the data will always be in a consistent way.
\graph ORMReplication02 ORM Cascaded Replication Classes via REST
subgraph cluster_0 {
label="Master";
\MasterServer¤(MasterModel)\master.db3
}
subgraph cluster_2 {
label="     Slave 1";
\MasterServer¤(MasterModel)\MasterClient¤(MasterModel)\HTTP
\MasterClient¤(MasterModel)\MasterServer¤(MasterModel)
\MasterClient¤(MasterModel)\SlaveServer1¤(SlaveModel1)\On Demand¤Replication
\SlaveServer1¤(SlaveModel1)\MasterClient¤(MasterModel)
\SlaveServer1¤(SlaveModel1)\slave1.db3
}
subgraph cluster_3 {
label="       Slave 2";
\SlaveServer1¤(SlaveModel1)\SlaveClient1¤(SlaveModel1)\HTTP
\SlaveClient1¤(SlaveModel1)\SlaveServer1¤(SlaveModel1)
\SlaveClient1¤(SlaveModel1)\SlaveServer2¤(SlaveModel2)\On Demand¤Replication
\SlaveServer2¤(SlaveModel2)\SlaveClient1¤(SlaveModel1)
\SlaveServer2¤(SlaveModel2)\slave2.db3
}
\
This cascading Master/Slave replication design may be used in conjunction with the @*CQRS@ pattern ({\i Command Query Responsibility Segregation}). In fact, the {\i Slave 2} database may be a local read-only database instance, used only for reporting purposes, e.g. by marketing or management people, whereas the {\i Slave 1} may be the active read-only database, on which all local business process will read their data. As such, the {\i Slave 2} instance may be replicated much less often than than {\i Slave 1} database - which may be even be replicated in real time, as we will now see.
:153  Real-time synchronization
Sometimes, the on-demand synchronization is not enough.
For instance, you may need to:
- Synchronize a short list of always evolving items which should be reflected as soon as possible;
- Involve some kind of ACID-like behavior (e.g. handle money!) in your replicated data;
- Replicate not from a GUI application, but from a service, so use of a {\f1\fs20 TTimer} is not an option;
- Combine REST requests (for ORM or services) and master/slave ORM replication on the same wire, e.g. in a multi-threaded application;
- Use an {\i @*Event Oriented Persistence@}, and expect to be notified from any change of state - see @171@.
In this case, the framework is able to use {\i WebSockets} and asynchronous callbacks to let the master/slave replication - see @149@ - take place without the need to ask explicitly for pending data. You will need to use {\f1\fs20 TSQLRestServer.RecordVersionSynchronizeMasterStart}, {\f1\fs20 TSQLRestServer.RecordVersionSynchronizeSlaveStart} and {\f1\fs20 TSQLRestServer.RecordVersionSynchronizeSlaveStop} methods over the proper kind of bidirectional connection.
The first requirement is to allow {\i WebSockets} on your {\i Master} HTTP server, so initialize the {\f1\fs20 TSQLHttpServer} class as a {\f1\fs20 useBidirSocket} kind of server - see @140@:
!  MasterServer := TSQLRestServerDB.Create(MasterModel,'master.db3');
!  HttpMasterServer := TSQLHttpServer.Create('8888',[MasterServer],'+',useBidirSocket);
!  HttpMasterServer.WebSocketsEnable(Server,'PrivateAESEncryptionKey');
On the {\i Slave} side, the HTTP client should also be upgraded to support {\i WebSockets}:
!  MasterClient := TSQLHttpClientWebsockets.Create('127.0.0.1',HTTP_DEFAULTPORT,MasterModel);
!  MasterClient.WebSocketsUpgrade('PrivateAESEncryptionKey');
Of course, the model should match for both {\f1\fs20 MasterServer} and {\f1\fs20 MasterClient} instances. As the {\i WebSockets} protocol definition - here above the same {\f1\fs20 'PrivateAESEncryptionKey'} private key.
Then you enable the real-time replication service on the {\i Master} side:
!  MasterServer.RecordVersionSynchronizeMasterStart;
In practice, it will publish a {\f1\fs20 IServiceRecordVersion} {\f1\fs20 interface}-based service on the server side - see @63@.
Assuming that the {\i slave} database has been defined as such:
!  SlaveServer := TSQLRestServerDB.Create(SlaveModel,'slave.db3');
(in this case, the {\f1\fs20 SlaveModel} may not be the same as the {\f1\fs20 MasterModel}, but {\f1\fs20 TSQLRecordPeopleVersioned} should be part of both models)\line Then you can initiate real-time replication from the {\i slave} side with a single line, for a given table:
!  SlaveServer.RecordVersionSynchronizeSlaveStart(TSQLRecordPeopleVersioned,MasterClient);
The above command will subscribe to the remote {\f1\fs20 MasterSlave} replication service (i.e. {\f1\fs20 IServiceRecordVersion} {\f1\fs20 interface}), to receive any change concerning the {\f1\fs20 TSQLRecordPeopleVersioned} ORM table, using the {\f1\fs20 MasterClient} connection via {\i WebSockets}, and persist all updates into the local {\f1\fs20 SlaveServer} database.
To stop the real-time notification for this ORM table, you could execute:
!  SlaveServer.RecordVersionSynchronizeSlaveStop(TSQLRecordPeopleVersioned);
Even if you do not call {\f1\fs20 RecordVersionSynchronizeSlaveStop()}, the replication will be stopped when the main {\f1\fs20 SlaveServer} instance will be released, and the {\f1\fs20 MasterServer} be {\i unsubscribe} this connection for its internal notification list.
This typical replication may be represented as such:
\graph ORMReplication1 ORM Real-Time Replication Classes
subgraph cluster_0 {
label="Master";
\MasterServer¤(MasterModel)\master.db3
}
subgraph cluster_2 {
label="       Slave";
\MasterServer¤(MasterModel)\MasterClient¤(MasterModel)\WebSockets¤TCP/IP
\MasterClient¤(MasterModel)\SlaveServer¤(SlaveModel)\Replication
\SlaveServer¤(SlaveModel)\slave.db3
}
\
The real-time notification details have been tuned, to consume as minimum bandwidth and resources as possible. For instance, if several modifications are to be notified on a slave connection in a short amount of time, the master is able to gather those modifications as a single {\i WebSockets} frame, which will be applied as a whole to the slave database, in a single BATCH transaction - see @28@.
:  Replication use cases
We may consider a very common corporate infrastructure:
\graph ORMReplication2 Corporate Servers Replication
subgraph cluster_0 {
label="Main Office";
\Main¤Server\External DB
}
subgraph cluster_1 {
label="           Office A";
\Main¤Server\Local¤Server A\HTTP
\Local¤Server A\Client 1
\Local¤Server A\Client 2
\Local¤Server A\Client 3\local¤network
}
subgraph cluster_2 {
label="          Office B";
\Main¤Server\Local¤Server B\HTTP
\Local¤Server B\Client  1
\Local¤Server B\Client  2
\Local¤Server B\Client  3
\Local¤Server B\Client  4\local¤network
}
\
This kind of installation, with a main central office, and a network of local offices, will benefit from this @*master/slave@ @*replication@. Simple {\i @*redirection@} may be used - see @93@ - but it will expect the work to continue, even in case of {\i Internet} network failure. REST redirection will expect a 100% connection up-link, which may be critical in some cases.
You could therefore implement replication in several ways:
- Either the main office is the master, and any write will be push to the {\i Main Server}, whereas local offices will have a replicated copy of the information - drawback is that in case of network failure, the local office will be limited to read only data access;
\graph ORMReplication3 Corporate Servers Master/Slave Replication With All Data On Main Server
subgraph cluster_0 {
label="Main Office";
\Main¤Server\Local Data A¤Read/Write
\Main¤Server\Local Data B¤Read/Write
}
subgraph cluster_1 {
label="           Office A";
\Local Data A¤Read Only\Main¤Server\Replication
\Local Data B¤Read Only\Main¤Server
}
subgraph cluster_2 {
label="          Office B";
\Local Data A¤ Read Only\Main¤Server\Replication
\Local Data B¤ Read Only\Main¤Server
}
\
- Or each local office may host its own data in a dedicated table, synchronized as a master database; the main office will replicate (as a slave) the private data of each local server; in addition, all this data gathered by the {\i Main Server} may be further replicated to the other local offices, and be still accessible in read mode - in case of network failure, all the data is available on local servers, and the local private table is still writable.
\graph ORMReplication4 Corporate Servers Master/Slave Replication With Private Local Data
subgraph cluster_0 {
label="Main Office";
\Main¤Server\Local Data A¤Reference¤Read Only
\Main¤Server\Local Data B¤Reference¤Read Only
}
subgraph cluster_1 {
label="           Office A";
\Local Data A¤Business¤Read/Write\Main¤Server\Replication
\Local Data B¤Business¤Read Only\Main¤Server
}
subgraph cluster_2 {
label="          Office B";
\Local Data A¤Business¤Read Only\Main¤Server\Replication
\Local Data B¤Business¤Read/Write\Main¤Server
}
\
Of course, the second solution seems preferable, even if a bit more difficult to implement. The ablity of all local offices to work offline on their own private data, but still having all the other data accessible as read-only, will be a huge ROI.
As a benefit of using replication, the central main server will be less stressed, since most of the process will take place in local servers, and the main office server will only be used for shared data backup and read-only gathering of the other local databases. Only a small network bandwith will be necessary (much less than a pure web solution), and CPU/storage resources will be minimal.
If needed, the @153@ will allow to have the main office data replicated in "near real-time" in the local offices databases, whereas the write operations will still safely take place on the main Office. Another cascading replication may take place within any node, with a on-demand refresh, e.g. a 1 hour period, to implement the @*CQRS@ pattern ({\i Command Query Responsibility Segregation}).
\graph ORMReplication5 Corporate Servers Master/Slave Replication With CQRS
subgraph cluster_0 {
label="Main Office";
\Main¤Server\Local Data A¤Reference¤Read Only
\Main¤Server\Local Data B¤Reference¤Read Only
}
subgraph cluster_1 {
label="          Office B";
\Local Data A¤Business¤Read Only\Main¤Server\RealTime¤Replication
\Local Data B¤Business¤Read/Write\Main¤Server
\Local Data B¤Reporting¤Read Only\Local Data B¤Business¤Read/Write\OnDemand¤Replication
\Local Data A¤Reporting¤Read Only\Local Data A¤Business¤Read Only\OnDemand¤Replication
}
\
Following the CQRS pattern, some demanding Queries may take place in those read-only "Reporting" replicated databases, without impacting the main local databases, in which all actual "Business" will take place.
:Daily ORM
%cartoon03.png
When you compare @*ORM@ and standard @*SQL@, some aspects must be highlighted.
First, you do not have to worry about field orders and names, and can use field completion in the IDE. It is much more convenient to type {\f1\fs20 Baby}. then select the {\f1\fs20 Name} property, and access to its value.
The ORM code is much more readable than the SQL. You do not have to switch your mind from one syntax to another, in your code. Because @*SQL@ is a true language (see {\i SQL Is A High-Level Scripting Language} at @http://www.fossil-scm.org/index.html/doc/tip/www/theory1.wiki ). You can even forget about the SQL itself for most projects; only some performance-related or complex queries should be written in SQL, but you will avoid it most of the time. Think object pascal. And happy coding. Your software architecture will thank you for it.
Another good impact is the naming consistency. For example, what about if you want to rename your table? Just change the class definition, and your IDE will do all refactoring for you, without any risk of missing a hidden SQL statement anywhere. Do you want to rename or delete a field? Change the class definition, and the {\i Delphi} compiler will let you know all places where this property was used in your code. Do you want to add a field to an existing database? Just add the property definition, and the framework will create the missing field in the database schema for you.
Another risk-related improvement is about the @**strong type@ checking, included into the {\i Delphi} language during compile time, and only during execution time for the SQL. You will avoid most runtime exceptions for your database access: your clients will thank you for that. In one word, forget about field typing mismatch or wrong type assignment in your database tables. Strong typing is great in such cases for code SQA, and if you worked with some scripting languages (like {\i @*JavaScript@}, Python or Ruby), you should have wished to have this feature in your project!
It is worth noting that our framework allows writing triggers and stored procedures (or like @*stored procedure@s) in {\i Delphi} code, and can create key indexing and perform foreign key checking in class definition.
Another interesting feature is the enhanced Grid component supplied with this framework, and the @*AJAX@-ready orientation, by using natively @*JSON@ flows for @*Client-Server@ data streaming. The @*REST@ protocol can be used in most application, since the framework provides you with an easy to use "Refresh" and caching mechanism. You can even work off line, with a local database @*replication@ of the remote data.
For Client-Server - see @6@ - you do not have to open a connection to the database, just create an instance of a {\f1\fs20 TSQLRestClient} object (with the communication layer you want to use: direct access, Windows Messages, named pipe or @*HTTP@), and use it as any normal {\i Delphi} object. All the @*SQL@ coding or communication and error handling will be done by the framework. The same code can be used in the Client or Server side: the parent {\f1\fs20 @*TSQLRest@} object is available on both sides, and its properties and methods are strong enough to access the data.
: ORM is not Database
It is worth emphasizing that you should not think about the @*ORM@ like a mapping of an existing DB schema. This is an usual mistake in ORM design.
The database is just one way of your objects persistence:
- Don't think about tables with simple types (text/number...), but objects with high level types;
- Don't think about @*Master/Detail@, but logical units;
- Don't think "@*SQL@", think about classes;
- Don't wonder "How will I store it?", but "Which data do I need?".
For instance, don't be tempted to always create a pivot table (via a {\f1\fs20 @*TSQLRecordMany@} property), but consider using a {\i @*dynamic array@}, {\f1\fs20 @*TPersistent@, @*TStrings@} or {\f1\fs20 @*TCollection@} @*published properties@ instead.
Or consider that you can use a {\f1\fs20 @*TRecordReference@} property pointing to any registered class of the {\f1\fs20 @*TSQLModel@}, instead of creating one {\f1\fs20 @*TSQLRecord@} property per potential table.
The {\i mORMot} framework is even able to persist the object without any SQL database, e.g. via {\f1\fs20 TSQLRestStorageInMemory}. In fact, its ORM core is optimized but not tied to SQL.
:  Objects, not tables
With an @*ORM@, you should usually define fewer tables than in a "regular" relational database, because you can use the high-level type of the {\f1\fs20 @*TSQLRecord@} properties to handle some per-row data.
The first point, which may be shocking for a database architect, is that you should better {\ul not} create @*Master/Detail@ tables, but just one "master" object with the details stored within, as @*JSON@, via {\i @*dynamic array@}, {\f1\fs20 @*TPersistent@, @*TStrings@} or {\f1\fs20 @*TCollection@} properties.
Another point is that a table is not to be created for every aspect of your software configuration. Let's confess that some DB architects design one configuration table per module or per data table. In an ORM, you could design a configuration class, then use the unique corresponding table to store all configuration encoded as some JSON data, or some DFM-like data. And do not hesitate to separate the configuration from the data, for all not data-related configuration - see e.g. how the {\f1\fs20 mORMotOptions} unit works. With our framework, you can serialize directly any {\f1\fs20 @*TSQLRecord@} or {\f1\fs20 TPersistent} instance into JSON, without the need of adding this {\f1\fs20 TSQLRecord} to the {\f1\fs20 @*TSQLModel@} list. Since revision 1.13 of the framework, you can even define {\f1\fs20 TPersistent} @published properties@ in your {\f1\fs20 TSQLRecord} class, and it will be automatically serialized as TEXT in the database.
:  Methods, not SQL
At first, you should be tempted to write code as such (this code sample was posted on our forum, and is not bad code, just not using the @*ORM@ orientation of the framework):
!  DrivesModel := CreateDrivesModel();
!  GlobalClient := TSQLRestClientDB.Create(DrivesModel, CreateDrivesModel(), 'drives.sqlite', TSQLRestServerDB);
!  TSQLRestClientDB(GlobalClient).Server.DB.Execute(
!    'CREATE TABLE IF NOT EXISTS drives ' +
!    '(id INTEGER PRIMARY KEY, drive TEXT NOT NULL UNIQUE COLLATE NOCASE);');
!  for X := 'A' to 'Z' do
!  begin
!    TSQLRestClientDB(GlobalClient).Server.DB.Execute(
!      'INSERT OR IGNORE INTO drives (drive) VALUES ("' + StringToUTF8(X) + ':")');
!  end;
Please, don't do that!
The correct ORM-oriented implementation should be the following:
!  DrivesModel := TSQLModel.Create([TDrives], 'root');
!  GlobalClient := TMyClient.Create(DrivesModel, nil, 'drives.sqlite', TSQLRestServerDB);
!!  GlobalClient.CreateMissingTables(0);
!!  if GlobalClient.TableRowCount(TDrives)=0 then
!  begin
!    D := TDrives.Create;
!    try
!      for X := 'A' to 'Z' do
!      begin
!        D.Drive := X;
!!        GlobalClient.Add(D,true);
!      end;
!    finally
!      D.Free;
!    end;
!  end;
In the above lines, no SQL was written. It is up to the @*ORM@ to:
- Create all missing tables, via the {\f1\fs20 CreateMissingTables} method - and not compute by hand a "{\f1\fs20 CREATE TABLE IF NOT EXISTS...}" SQL statement;
- Check if there is some rows of data, via the {\f1\fs20 TableRowCount} method - instead of a "{\f1\fs20 SELECT COUNT(*) FROM DRIVES}";
- Append some data using an high-level {\f1\fs20 TDrives} {\i Delphi} instance and the {\f1\fs20 Add} method - and not any "{\f1\fs20 INSERT OR IGNORE INTO DRIVES...}".
Then, in order to retrieve some data, you'll be tempted to code something like that (extracted from the same forum article):
!procedure TMyClient.FillDrives(aList: TStrings);
!var
!  table: TSQLTableJSON;
!  X, FieldIndex: Integer;
!begin
!  table := TSQLRestClientDB(GlobalClient).ExecuteList([TSQLDrives], 'SELECT * FROM drives');
!  if (table <> nil) then
!  try
!    FieldIndex := table.FieldIndex('drive');
!    if (FieldIndex >= 0) then
!      for X := 1 to table.RowCount do
!        Items.Add(UTF8ToString(table.GetU(X, FieldIndex)));
!  finally
!    table.Free;
!  end;
!end;
Thanks to the {\f1\fs20 TSQLTableJSON} class, code is somewhat easy to follow. Using a temporary {\f1\fs20 FieldIndex} variable make also it fast inside the loop execution.
But it could also be coded as such, using the {\f1\fs20 @*CreateAndFillPrepare@} then {\f1\fs20 FillOne} method in a loop:
!procedure TMyClient.FillDrives(aList: TStrings);
!begin
!  aList.BeginUpdate;
!  try
!    aList.Clear;
!!    with TSQLDrives.CreateAndFillPrepare(GlobalClient,'') do
!    try
!!      while FillOne do
!!        aList.Add(UTF8ToString(Drive));
!    finally
!      Free;
!    end;
!  finally
!    aList.EndUpdate;
!  end;
!end;
We even added the {\f1\fs20 BeginUpdate / EndUpdate} VCL methods, to have even cleaner and faster code (if you work with a {\f1\fs20 TListBox} e.g.).
Note that in the above code, an hidden {\f1\fs20 TSQLTableJSON} class is created in order to retrieve the data from the server. The abstraction introduced by the ORM methods makes the code not slowest, but less error-prone (e.g. {\f1\fs20 Drive} is now a {\f1\fs20 @*RawUTF8@} property), and easier to understand.
But @*ORM@ is not perfect in all cases.
For instance, if the {\f1\fs20 Drive} field is the only column content to retrieve, it could make sense to ask only for this very column. One drawback of the {\f1\fs20 CreateAndFillPrepare} method is that, by default, it retrieves all columns content from the server, even if you need only one. This is a common potential issue of an ORM: since the library doesn't know which data is needed, it will retrieve all object data, which is some cases is not worth it.
You can specify the optional {\f1\fs20 aCustomFieldsCSV} parameter as such, in order to retrieve only the {\f1\fs20 Drive} property content, and potentially save some bandwidth:
!!    with TSQLDrives.CreateAndFillPrepare(GlobalClient,'','Drive') do
Note that for this particular case, you have an even more high-level method, handling directly a {\f1\fs20 TStrings} property as the recipient:
!procedure TMyClient.FillDrives(aList: TStrings);
! begin
!!  GlobalClients.OneFieldValues(TSQLDrives,'drive','',aList);
! end;
The whole query is made in one line, with no {\f1\fs20 SELECT} statement to write.
For a particular ID range, you may have written, with a specific WHERE clause using a @*prepared@ statement:
!!  GlobalClients.OneFieldValues(TSQLDrives,'drive',
!!    'ID>=? AND ID<=?',[],[aFirstID,aLastID],aList);
It is certainly worth reading all the (verbose) interface part of the {\f1\fs20 mORMot.pas} unit, e.g. the {\f1\fs20 TSQLRest} class, to make your own idea about all the high-level methods available. In the following pages, you'll find all needed documentation about this particular unit. Since our framework is used in real applications, most useful methods should already have been made available. If you need additional high-level features, feel free to ask for them, if possible with source code sample, in our forum, freely available at @https://synopse.info
:  Think multi-tier
And do not forget the framework is able to have several level of objects, thanks to our @*Client-Server@ architecture - see @6@. Such usage is not only possible, but strongly encouraged.
You should have business-logic level objects at the Client side. Then both business-logic and DB objects at the Server side.
If you have a very specific database schema, business-logic objects can be of very high level, encapsulating some @*SQL@ views for reading, and accessed via some @*REST@ful @*service@ commands for writing - see @11@.
Another possibility to access your high-level type, is to use either custom {\i @*SQLite3@} @*SQL function@s or @*stored procedure@s - see @23@ - both coded in {\i Delphi}.
:168 One ORM to rule them all
Just before entering deeper into the {\i mORMot} material in the following pages (Database layer, Client-Server, Services), you may find out that this implementation may sounds restricted.
Some common (and founded) criticisms are the following (quoting from our forum):
- "One of the things I don't like so much about your approach to the @*ORM@ is the mis-use of existing {\i Delphi} constructs like "{\f1\fs20 @*index@ n}" attribute for the maximum length of a string-property. Other ORMs solve this i.e. with official {\f1\fs20 Class}-attributes";
- "You have to inherit from {\f1\fs20 TSQLRecord}, and can't persist any plain class";
- "There is no way to easily map an existing complex database".
Those concerns are pretty understandable. Our {\i mORMot} framework is not meant to fit any purpose, but it is worth understanding why it has been implemented as such, and why it may be quite unique within the family of ORMs - which almost all are following the {\i Hibernate} way of doing.
:  Rude class definition
Attributes do appear in {\i Delphi} 2010, and it is worth saying that @*FPC@ has an alternative syntax. Older versions of {\i Delphi} (still very deployed) do not have attributes available in the language, so it was not possible to be compatible with {\i Delphi} 6 up to latest versions (as we wished for our units).
It is perfectly right to speak about 'mis-use of {\f1\fs20 index}' - but this was the easiest and only way we found out to have such information, just using RTTI. Since this parameter was ignored and not used for most classes, it was re-used (also for dynamic array properties, to have faster lookup).\line There is another "mis-use" for the "{\f1\fs20 stored @*AS_UNIQUE@}" property, which is used to identify unique mandatory columns.
Using attributes is one of the most common ways of describing tables in most ORMs.\line On the other hand, some coders have a concern about such class definitions. They are mixing DB and logic: you are somewhat polluting the business-level class definition with DB-related stuff.
That is why other kind of ORMs provide a way of mapping classes to tables using external files (some ORMs provide both ways of definition). And why those days, even code gurus identified the attributes overuse as a potential weakness of code maintainability.\line Attributes do have a huge downsize, when you are dealing with a Client-Server ORM, like ours: on the Client side, those attributes are pointless (client does not need to know anything about the database), and you need to link to all the DB plumbing code to your application. For {\i mORMot}, it was some kind of strong argument.
For the very same reasons, the column definitions (uniqueness, indexes, required) are managed in {\i mORMot} at two levels:
- At {\i ORM level} for {\i DB related stuff} (like indexes, which is a DB feature, not a business feature);
- At {\i Model level} for {\i Business related stuff} (like uniqueness, validators and filters).
When you take a look at the supplied validators and filters - see @56@ - you'll find out that this is much powerful than the attributes available in "classic" ORMs: how could you validate an entry to be an email, or to match a pattern, or to ensure that it will be stored in uppercase within the DB?
Other question worth asking is about the security. If you access the data remotely, a global access to the DB is certainly not enough. Our framework handle per-table @*CRUD@ level access for its ORM, above the DB layer (and has also complete security attributes for services) - see @43@. It works however the underneath DB grants are defined (even an DB with no user rights - like in-memory or {\i SQLite3} is able to do it).
The {\i mORMot} point of view (which is not the only one), is to let the DB persist the data, as safe and efficient as possible, but rely on higher levels layers to implement the business logic. The framework favors {\i @*convention over configuration@}, which is known to save a lot of time (if you use WCF on a daily basis, as I do, you and your support team know about the {\f1\fs20 .config} syndrome). It will make it pretty database-agnostic (you can even not use a SQL database at all), and will make the framework code easier to debug and maintain, since we don't have to deal with all the DB engine particularities. In short, this is the @*REST@ point of view, and main cause of success: @*CRUD@ is enough in any @*KISS@-friendly design.
:  Persist TSQLRecord, not any class
About the fact that you need to inherit from {\f1\fs20 @*TSQLRecord@}, and can't persist any @*PODO@ ({\i Plain Old Delphi Object}), our purpose was in fact very similar to the "@**Layer Supertype@" pattern of @*Domain-Driven@-Design, as explained by Martin Fowler:\line {\i It is not uncommon for all the objects in a layer to have methods you don't want to have duplicated throughout the system. You can move all of this behavior into a common Layer Supertype.}
In fact, for {\f1\fs20 TSQLRecord} / {\f1\fs20 TSQLRest} / @*ORM@ remote access, you have already all @*Client-Server@ @*CRUD@ operations available. Those classes are abstract common Supertypes, ready to be used in your projects. It has been optimized a lot (e.g. with a cache and other nice features), so I do not think reinventing a CRUD / database service is worth the prize. You have secure access to the ORM classes, with user/group attributes. Almost everything is created by code, just from the {\f1\fs20 TSQLRecord} class definition, via @*RTTI@. So it may be faster (and safer) to rely on it, than defining all your class hierarchy by hand.
To be fair, most DDD frameworks for Java or C# expect e.g. {\i Entities} classes to inherit from a given {\f1\fs20 Entity} class, or add {\i class attributes} to the POJO/POCO to define the persistence details. So we are not the single one in this case!
But the concern of not being able to persist any class (it needs to inherit from {\f1\fs20 TSQLRecord}) does perfectly make sense. Especially in the context of DDD modeling, where the DDD objects will benefit from being uncoupled from the framework, which may pollute the domain logic. All those expectations tend to break the {\i @*Persistence Ignorance@} principle, as requested by DDD patterns.
This is why we added to the framework the ability to persist any plain class, using Repository services, but still using the ORM under the hood, for the actual persistence on any SQL or NoSQL database engine. The {\f1\fs20 TSQLRecord} can be generated from any Delphi persistent {\f1\fs20 class}, then an automated mapping is maintained by {\i mORMot} between both {\i class} instances. Data access is then defined as clean @*CQRS@ {\i Repository Services}.\line For instance, a {\f1\fs20 TUser} class may be persisted via such a service:
!type
!  IDomUserCommand = interface(IDomUserQuery)
!    ['{D345854F-7337-4006-B324-5D635FBED312}']
!    function Add(const aAggregate: TUser): TCQRSResult;
!    function Update(const aUpdatedAggregate: TUser): TCQRSResult;
!    function Delete: TCQRSResult;
!    function Commit: TCQRSResult;
!  end;
Here, the write operations are defined in a {\f1\fs20 IDomUserCommand} service, which is separated (but inherits) from {\f1\fs20 IDomUserQuery}, which is used for read operations. See @167@ for more details about this feature.
:  Several ORMs at once
To be clear, {\i mORMot} offers several kind of table definitions:
- Via {\f1\fs20 @*TSQLRecord@} / {\f1\fs20 @*TSQLRecordVirtual@} "native ORM" classes: data storage is using either fast in-memory lists via {\f1\fs20 @*TSQLRestStorageInMemory@}, or {\i @*SQLite3@} tables (in memory, on file, or virtual). In this case, we do not use {\f1\fs20 index} for strings (column length is not used by any of those engines).
- Via {\f1\fs20 @*TSQLRecord@} "external ORM-managed" classes: after registration via a call to the {\f1\fs20 @*VirtualTableExternalRegister@()} / {\f1\fs20 @*VirtualTableExternalMap@()} functions, external DB tables are created and managed by the ORM, via SQL - see @27@. These classes will allow creation of tables in any supported database engine - currently {\i SQLite3, @*Oracle@, @*Jet/MSAccess@, @*MS SQL@, @*Firebird@, @*DB2@, @*PostgreSQL@, @*MySQL@, @*Informix@} and {\i @*NexusDB@} - via whatever {\i OleDB, ODBC} / @*ZDBC@ provider, or any {\f1\fs20 DB.pas} unit). For the "external ORM-managed" {\f1\fs20 TSQLRecord} type definitions, the ORM expects to find an {\f1\fs20 index} attribute for any text column length (i.e. {\f1\fs20 RawUTF8} or {\f1\fs20 string} published properties). This is the only needed parameter to be defined for such a basic implementation, in regard to {\f1\fs20 TSQLRecord} kind of classes. Then can specify addition field/column mapping, if needed.
- Via {\f1\fs20 @*TSQLRecord@} "external @*ODM@-managed" classes: after registration via a call to the {\f1\fs20 @*StaticMongoDBRegister@()} or {\f1\fs20 @*StaticMongoDBRegisterAll@()} functions, external {\i @*MongoDB@} collections are created and managed via @82@. In this case, no {\f1\fs20 index} attribute for setting text column length is necessary.
- Via {\f1\fs20 @*TSQLRecordMappedAutoID@} / {\f1\fs20 @*TSQLRecordMappedForcedID@} "external mapped" classes: DB tables are not created by the ORM, but already existing in the DB, with sometimes a very complex layout. This feature is not yet implemented, but on the road-map. For this kind of classes we won't probably use attributes, nor even external files, but we will rely on definition from code, either with a fluent definition, or with dedicated classes (or interface).
- Via any kind of Delphi {\f1\fs20 class}, mapped to their internal {\f1\fs20 TSQLRecord class}, using @*CQRS@ {\i Repository Services} as presented @167@.
Why have several database back-end at the same time?
Most of the existing software architecture rely on one dedicated database per domain, since it is more convenient to administrate one single server.\line But there are some cases when it does make sense to have several databases at once.
In practice, when your data starts to grow, you may need to {\i archive} older data in a dedicated remote database, e.g. using cheap storage (bunch of Hard Drives in RAID). Since this data will be seldom retrieved, it is not an issue to have slower access time. And you will be able to keep your most recent data accessible in a local high-speed engine (running on SSD).
Another pattern is to use dedicated consolidation DBs for any analysis. In fact, SQL {\i @*normalization@} is good for most common relation work, but sometimes {\i @*denormalization@} is necessary, e.g. for statistic or business analyse purposes. In this case, dedicated consolidation databases, containing the data already prepared and indexed in a ready-to-use denormalized layout.
Last but not least, some @*Event Sourcing@ architectures even {\i expect} several DB back-end at once:
- It will store the status on one database (e.g. high-performance in-memory) for most common requests to be immediate;
- And store the modification events in another @*ACID@ database (e.g. {\i SQLite3, @*Oracle@, @*Jet/MSAccess@, @*MS SQL@, @*Firebird@, @*DB2@, @*PostgreSQL@, @*MySQL@, @*Informix@} or {\i @*NexusDB@}), even a high-speed @*NoSQL@ engine like {\i @*MongoDB@}.
It is possible to directly access ORM objects remotely (e.g. the consolidation DB), mostly in a read-only way, for dedicated reporting, e.g. from consolidated data - this is one potential @*CQRS@ implementation pattern with {\i mORMot}. Thanks to the framework security, remote access will be safe: your clients won't be able to change the consolidation DB content!
As can be easily guessed, such design models are far away from a basic ORM built only for class persistence. And {\i mORMot}'s ORM/ODM offers you all those possibilities.
:  The best ORM is the one you need
Therefore, we may sum up some potential use of ORM, depending of your intent:
- If you want to persist some data objects (not tied to complex business logic), the framework's ORM will be a light and fast candidate, targetting {\i SQLite3, @*Oracle@, @*Jet/MSAccess@, @*MS SQL@, @*Firebird@, @*DB2@, @*PostgreSQL@, @*MySQL@, @*Informix@, @*NexusDB@} databases, or even with no SQL engine, using {\f1\fs20 @*TSQLRestStorageInMemory@} class which is able to persist its content in small files - see @57@;
- If your understanding of ORM is just to persist some {\i existing} objects with associated business code, {\i mORMot} could help you, thanks to its {\i Repository Services} automatically generated over {\f1\fs20 TSQLRecord}, as presented @167@;
- If you want a very fast low-level Client-Server layer, {\i mORMot} is a first class candidate: we identified that some users are using the built-in JSON serialization and HTTP server features to create their application, using a RESTful/SOA architecture - see @49@ and @63@;
- If your expectation is to map an existing complex RDBMS, {\i mORMot} will allow to publish existing SQL statements as services, using e.g. {\f1\fs20 interface}-based services - see @63@ - over optimized {\f1\fs20 SynDB.pas} data access - see @126@ - as explained in @66@;
- If you need (perhaps not now, but probably in the future) to create some kind of scalable {\i @*Domain-Driven@} design architecture, you'll have all needed features at hand with {\i mORMot};
Therefore, {\i mORMot} is not just an ORM, nor just a "classic" ORM.
:42Database layer
%cartoon04.png
: SQLite3-powered, not SQLite3-limited
The core database of this framework uses the {\i @**SQLite3@} library, which is a Free, Secure, Zero-Configuration, Server-less, Single Stable Cross-Platform Database File database engine.
As stated below, you can use any other database access layer, if you wish:
- A fast in-memory engine is included, which outperforms any SQL-based solution in terms of speed - but to the price of a non ACID behavior on disk (but ACID in RAM);
- An integrated {\i SQLite3} engine, which is the best candidate for an embedded solution, even on server side;
- Any remote RDBMS database, via one or more {\i @*OleDB@}, {\i @*ODBC@}, {\i @*Zeos@} or {\i @*Oracle@} connections to store your precious ORM objects. Or you can use any {\f1\fs20 DB.pas} unit, e.g. to access @*NexusDB@ or any database engines supported by @*DBExpress@, @*FireDAC@, @*AnyDAC@, @*UniDAC@ (or the deprecated @*BDE@). In all cases, the ORM supports currently {\i SQLite3, @*Oracle@, @*Jet/MSAccess@, @*MS SQL@, @*Firebird@, @*DB2@, @*PostgreSQL@, @*MySQL@, @*Informix@} and {\i @*NexusDB@} SQL dialects;
- Any other {\f1\fs20 TSQLRest} instance (either another {\f1\fs20 TSQLRestServer}, or a remote {\f1\fs20 TSQLRestClientHTTP}) - see @93@;
- Direct access to a {\i @*MongoDB@} database, which implements a true @82@ design.
\graph mORMotDBDesign mORMot Persistence Layer Architecture
\ORM\SQLite3\direct
\ORM\TObjectList¤NoSQL\direct
\ORM\RDBMS\direct
\SQLite3\TObjectList¤NoSQL\virtual
\SQLite3\RDBMS
\RDBMS\Oracle SQLite3 ODBC¤OleDB ZDBC\direct
\RDBMS\FireDAC AnyDAC UniDAC¤BDE DBExpress NexusDB\DB.pas¤TDataSet
\ORM\MongoDB¤NoSQL\direct
\ORM\TSQLRest¤redirection\direct
=ORM=mORMot¤ORM
=RDBMS=External SQL¤RDBMS
\TObjectList¤NoSQL=RDBMS=MongoDB¤NoSQL=TSQLRest¤redirection
\
{\i SQlite3} will be used as the main SQL engine, able to @*JOIN@ all those tables, thanks to its {\i @*Virtual Table@} unique feature. You can in fact {\i mix} internal and external engines, in the same database model, and access all data in one unique SQL statement.
:  SQLite3 as core
This framework uses a compiled version of the official {\i SQLite3} library source code, and includes it natively into {\i Delphi} code. This framework therefore adds some very useful capabilities to the Standard {\i SQLite3} database engine, but keeping all its advantages, as listed in the previous paragraph of this document:
- Can be either statically linked to the executable, or load external {\f1\fs20 sqlite3.dll};
- Faster database access, through unified memory model, and usage of the {\f1\fs20 @*FastMM4@} memory manager (which is almost 10 times faster than the default Windows memory manager for memory allocation);
- Optional direct @*encryption@ of the data on the disk (up to @*AES@256 level, that is Top-Secret @*security@);
- Use via {\i mORMot}'s @*ORM@ let database layout be declared once in the {\i Delphi} source code (as @*published properties@ of classes), avoiding most SQL writing, hence common field or table names mismatch;
- Locking of the database at the record level ({\i SQLite3} only handles file-level locking);
- Of course, the main enhancement added to the {\i SQLite3} engine is that it can be deployed in a @*stand-alone@ or @*Client-Server@ architecture, whereas the default {\i SQLite3} library works only in stand-alone mode.
From the technical point of view, here are the current compilation options used for building the {\i SQLite3} engine:
- Uses @*ISO 8601@:2004 format to properly handle date/time values in TEXT field, or in faster and smaller {\f1\fs20 Int64} custom types: {\f1\fs20 @*TTimeLog@ / @*TModTime@ / @*TCreateTime@} or {\f1\fs20 @*TUnixTime@} / {\f1\fs20 @*TUnixMSTime@};
- {\i SQLite3} library unit was compiled including @*RTREE@ extension for doing very fast range queries;
- It can include @*FTS@3/FTS4 @*full text@ search engine (MATCH operator), with integrated @*SQL@ optimized ranking function;
- The framework makes use only of newest API ({\f1\fs20 sqlite3_prepare_v2}) and follows latest {\i SQLite3} official documentation;
- Additional {\i @*collation@s} (i.e. sorting functions) were added to handle efficiently not only @*UTF-8@ text, but also e.g. @*ISO 8601@ time encoding, fast {\i Win1252} diacritic-agnostic comparison and native slower but accurate Windows UTF-16 functions;
- Additional @*SQL@ functions like {\i Soundex} for English/French/Spanish phonemes, {\f1\fs20 MOD} or {\f1\fs20 CONCAT}, and some dedicated functions able to directly search for data within @*BLOB@ fields containing an {\i Delphi} high-level type (like a serialized dynamic array);
- Additional {\f1\fs20 REGEXP} operator/function using the Open Source PCRE library to perform @*regular expression@ queries in SQL statements;
- Custom @*SQL@ functions can be defined in {\i Delphi} code;
- Automatic SQL statement parameter preparation, for execution speed up;
- {\f1\fs20 TSQLDatabase} can cache the last results for SELECT statements, or use a tuned client-side or server-side per-record caching, in order to speed up most read queries, for lighter web server or client User Interface e.g.;
- User @*authentication@ handling ({\i SQLite3} is user-free designed);
- {\i SQLite3} source code was compiled without thread mutex: the caller has to be @*thread-safe@ aware; this is faster on most configurations, since mutex has to be acquired once): low level {\f1\fs20 sqlite3_*()} functions are not thread-safe, as {\f1\fs20 TSQLRequest} and {\f1\fs20 TSQLBlobStream} which just wrap them; but {\f1\fs20 TSQLDataBase} is thread-safe, as {\f1\fs20 TSQLTableDB}/{\f1\fs20 @*TSQLRestServerDB@}/{\f1\fs20 TSQLRestClientDB} which call {\f1\fs20 TSQLDataBase};
- Compiled with {\f1\fs20 SQLITE_OMIT_SHARED_CACHE} define, since with the new @*Client-Server@ approach of this framework, no concurrent access could happen, and an internal efficient caching algorithm is added, avoiding most call of the {\i SQLite3} engine in multi-user environment (any @*AJAX@ usage should benefit of it);
- The embedded {\i SQLite3} database engine can be easily updated from the official {\i SQLite3} source code available at @https://sqlite.org - use the amalgamation C file with a few minor changes (documented in the {\f1\fs20 SynSQLite3Static.pas} unit) - the resulting C source code delivered as {\f1\fs20 .obj/.o} is also available in the official {\i Synopse} source code repository.
The overhead of including {\i SQlite3} in your server application will be worth it: just around 1 MB to the executable, but with so many nice features, even if only external databases are used.
:  Extended by SQLite3 virtual tables
Since the framework is truly object oriented, another database engine could be used instead of the framework. You could easily write your own {\f1\fs20 TSQLRestServer} descendant (as an example, we included a fast in-memory database engine as {\f1\fs20 @*TSQLRestServerFullMemory@}) and link to a another engine (like {\i @*FireBird@}, or a private one). You can even use our framework without any link to the {\i @*SQLite3@} engine itself, via our provided very fast in memory dataset (which can be made persistent by writing and reading @*JSON@ files on disk). The {\i SQLite3} engine is implemented in a separate unit, named {\f1\fs20 SynSQLite3.pas}, and the main unit of the framework is {\f1\fs20 mORMot.pas}. A bridge between the two units is made with {\f1\fs20 mORMotSQLite3.pas}, which will found our ORM framework using {\i SQLite3} as its core.
The framework ORM is able to access any database class (internal or external), via the powerful {\i SQLite3} Virtual Table mechanisms - see @20@. For instance, any external database (via @*OleDB@ / @*ODBC@ / @*ZDBC@ providers or direct {\i @*Oracle@} connection) can be accessed via our {\f1\fs20 @*SynDB@.pas}-based dedicated units, as stated @27@.
As a result, the framework has several potential database back-ends, in addition to the default {\i SQLite3} file-based engine. Each engine may have its own purpose, according to the application expectations. Currently {\i SQLite3, Oracle, @*Jet/MSAccess@, @*MS SQL@, @*Firebird@, @*DB2@, @*PostgreSQL@, @*MySQL@, @*Informix@} and {\i @*NexusDB@} SQL dialects are handled by our ORM.
:59  Data access benchmark
Purpose here is not to say that one library or database is better or faster than another, but publish a snapshot of {\i mORMot} persistence layer abilities, depending on each access library.
In this timing, we do not benchmark only the "pure" SQL/DB layer access ({\f1\fs20 SynDB.pas} units), but the whole @*Client-Server@ ORM of our framework.
Process below includes all aspects of our ORM:
- Access via high level @*CRUD@ methods ({\i Add/Update/Delete/Retrieve}, either per-object or in BATCH mode);
- Read and write access of {\f1\fs20 TSQLRecord} instances, via optimized RTTI;
- @*JSON@ marshalling of all values (ready to be transmitted over a network);
- @*REST@ routing, with security, logging and statistic;
- Virtual cross-database layer using its {\i SQLite3} kernel;
- SQL on-the-fly generation and translation (in {\i virtual} mode);
- Access to the database engines via several libraries or providers.
In those tests, we just bypassed the communication layer, since {\f1\fs20 TSQLRestClient} and {\f1\fs20 TSQLRestServer} are run in-process, in the same thread - as a {\f1\fs20 @*TSQLRestServerDB@} instance. So you have here some raw performance testimony of our framework's ORM and RESTful core, and may expect good scaling abilities when running on high-end hardware, over a network.
On a recent notebook computer ({\i Core i7} and SSD drive), depending on the back-end database interfaced, {\i mORMot} excels in speed, as will show the following @**benchmark@:
- You can persist up to 570,000 objects per second, or retrieve 870,000 objects per second (for our pure {\i Delphi} in-memory engine);
- When data is retrieved from server or client @38@, you can read more than 900,000 objects per second, whatever the database back-end is;
- With a high-performance database like @*Oracle@, and our direct access classes, you can write 70,000 (via @*array bind@ing) and read 160,000 objects per second, over a 100 MB network;
- When using alternate database access libraries (e.g. @*Zeos@, or {\f1\fs20 DB.pas} based classes), speed is lower (even if comparable for @*DB2@, @*MS SQL@, @*PostgreSQL@, @*MySQL@) but still enough for most work, due to some optimizations in the {\i mORMot} code (e.g. caching of prepared statements, SQL multi-values insertion, direct export to/from JSON, {\i SQlite3} virtual mode design, avoid most temporary memory allocation...).
Difficult to find a faster ORM, I suspect.
:   Software and hardware configuration
The following tables try to sum up all available possibilities, and give some benchmark (average objects/second for writing or reading).
In these tables:
- 'SQLite3 (file full/off/exc)' indicates use of the internal {\i @*SQLite3@} engine, with or without {\f1\fs20 Synchronous := smOff} and/or {\f1\fs20 DB.LockingMode := lmExclusive} - see @60@;
- 'SQLite3 (mem)' stands for the internal {\i SQLite3} engine running in memory;
- 'SQLite3 (ext ...)' is about access to a {\i SQLite3} engine as external database - see @27@, either as file or memory;
- '{\f1\fs20 TObjectList}' indicates a {\f1\fs20 TSQLRestStorageInMemory} instance - see @57@ - either static (with no SQL support) or virtual (i.e. SQL featured via {\i SQLite3} virtual table mechanism) which may persist the data on disk as JSON or compressed binary;
- 'WinHTTP SQLite3' and 'Sockets SQLite3' stands for a {\i SQLite3} engine published over HTTP using our {\f1\fs20 SynDBRemote.pas} unit using the {\f1\fs20 WinHTTP} API or plain sockets on the client side - see @131@, then accessed as an external database by our ORM;
- '@*NexusDB@' is the free embedded edition, available from official site;
- 'Jet' stands for a {\i @*Jet/MSAccess@} database engine, accessed via OleDB.
- 'Oracle' shows the results of our direct OCI access layer ({\f1\fs20 SynDBOracle.pas});
- '@*Zeos@ *' indicates that the database was accessed directly via the @*ZDBC@ layer;
- 'FireDAC *' stands for {\i @*FireDAC@} library;
- 'UniDAC *' stands for {\i @*UniDAC@} library;
- 'BDE *' when using a {\i @*BDE@} connection;
- '@*ODBC@ *' for a direct access to ODBC;
- '@*MongoDB@ ack/no ack' for direct {\i MongoDB} access ({\f1\fs20 SynMongoDB.pas}) with or without write acknowledge.
This list of database providers is to be extended in the future. Any feedback is welcome!
Numbers are expressed in rows/second (or objects/second). This benchmark was compiled with {\i Delphi} XE4, since newer compilers tends to give better results, mainly thanks to function in-lining (which was not existing e.g. in {\i Delphi} 6-7).
Note that these tests are not about the relative speed of each database engine, but reflect the current status of the integration of several DB libraries within the {\i mORMot} database access.
Benchmark was run on a {\i Core i7} notebook, running {\i Windows 7}, with a standard SSD, including anti-virus and background applications:
- Linked to a shared {\i @*Oracle@} 11.2.0.1 database over 100 Mb Ethernet;
- {\i @*MS SQL@ Express 2008 R2} running locally in 64-bit mode;
- {\i IBM @*DB2@ Express-C} edition 10.5 running locally in 64-bit mode;
- {\i @*PostgreSQL@} 9.2.7 running locally in 64-bit mode;
- {\i @*MySQL@} 5.6.16 running locally in 64-bit mode;
- {\i @*Firebird@} embedded in revision 2.5.2;
- {\i @*NexusDB@} 3.11 in Free Embedded Version;
- {\i MongoDB} 2.6 in 64-bit mode.
So it was a development environment, very similar to low-cost production site, not dedicated to give best performance. During the process, CPU was noticeable used only for {\i SQLite3} in-memory and {\i TObjectList} - most of the time, the bottleneck is not the CPU, but the storage or network. As a result, rates and timing may vary depending on network and server load, but you get results similar to what could be expected on customer side, with an average hardware configuration. When using high-head servers and storage, running on a tuned {\i @*Linux@} configuration, you can expect even better numbers.
Tests were compiled with the {\i Delphi} XE4 32-bit mode target platform. Most of the tests do pass when compiled as a 64-bit executable, with the exception of some providers (like Jet), not available on this platform. Speed results are almost the same, only slightly slower; so we won't show them here.
You can compile the "{\f1\fs20 15 - External DB performance}" supplied sample code, and run the very same benchmark on your own configuration. Feedback is welcome!
From our tests, the UniDAC version we were using had huge stability issues when used with DB2: the tests did not pass, and the DB2 server just hang processing the queries, whereas there was no problem with other libraries. It may have been fixed since, but you won't find any "UniDAC DB2" results in the benchmark below in the meanwhile.
:   Insertion speed
Here we insert 5,000 rows of data, with diverse scenarios:
- 'Direct' stands for a individual {\f1\fs20 Client.Add()} insertion;
- 'Batch' mode will be described @28@;
- 'Trans' indicates that all insertion is nested within a transaction - which makes a great difference, e.g. with a {\i SQlite3} database.
Here are some insertion speed values, in objects/second:
|%30%15%15%15%15
||\b Direct|Batch|Trans|Batch Trans\b0
|{\b SQLite3 (file full)}|462|28123|84823|181455
|{\b SQLite3 (file off)}|2102|83093|88006|202667
|{\b SQLite3 (file off exc)}|28847|193453|89451|207615
|{\b SQLite3 (mem)}|89456|236540|104249|239165
|{\b TObjectList (static)}|314465|543892|326370|542652
|{\b TObjectList (virtual)}|325393|545672|298846|545018
|{\b SQLite3 (ext full)}|424|14523|102049|164636
|{\b SQLite3 (ext off)}|2245|47961|109706|189250
|{\b SQLite3 (ext off exc)}|41589|180759|108481|192071
|{\b SQLite3 (ext mem)}|101440|211389|113530|209713
|{\b WinHTTP SQLite3}|2165|36464|2079|38478
|{\b Sockets SQLite3}|8118|75251|8553|80550
|{\b MongoDB (ack)}|10081|84585|9800|85232
|{\b MongoDB (no ack)}|33223|273672|34665|274393
|{\b ODBC SQLite3}|492|11746|35367|82425
|{\b ZEOS SQlite3}|494|11851|56206|85705
|{\b FireDAC SQlite3}|20605|38853|40042|113752
|{\b UniDAC SQlite3}|477|8725|26552|38756
|{\b ODBC Firebird}|1495|18056|13485|17731
|{\b ZEOS Firebird}|10452|62851|22003|63708
|{\b FireDAC Firebird}|18147|46877|18922|46353
|{\b UniDAC Firebird}|5986|14809|6522|14948
|{\b Jet}|4235|4424|4954|5094
|{\b NexusDB}|5998|15494|7687|18619
|{\b Oracle}|226|56112|1133|52367
|{\b ZEOS Oracle}|210|32725|1027|31982
|{\b ODBC Oracle}|236|1664|1515|7709
|{\b FireDAC Oracle}|118|48575|1519|12566
|{\b UniDAC Oracle}|164|5701|1215|2884
|{\b BDE Oracle}|489|927|839|1022
|{\b MSSQL local}|5246|54360|12988|62453
|{\b ODBC MSSQL}|4911|18652|11541|20976
|{\b FireDAC MSSQL}|5016|7341|11686|51242
|{\b UniDAC MSSQL}|4392|29768|8649|33464
|{\b ODBC DB2}|4792|48387|14085|70104
|{\b FireDAC DB2}|4452|48635|11014|52781
|{\b ZEOS PostgreSQL}|4196|31409|9689|41225
|{\b ODBC PostgreSQL}|4068|26262|5130|30435
|{\b FireDAC PostgreSQL}|4181|26635|10111|36483
|{\b UniDAC PostgreSQL}|2705|18563|4442|28337
|{\b ODBC MySQL}|3160|38309|10856|47630
|{\b ZEOS MySQL}|3426|34037|12217|40186
|{\b FireDAC MySQL}|3078|43053|10955|45781
|{\b UniDAC MySQL}|3119|27772|11246|33288
|%
Due to its ACID implementation, {\i SQLite3} process on file waits for the hard-disk to have finished flushing its data, therefore it is the reason why it is slower than other engines at individual row insertion (less than 10 objects per second with a mechanical hard drive instead of a SDD) outside the scope of a transaction.
So if you want to reach the best writing performance in your application with the default engine, you should better use transactions and regroup all writing into services or a BATCH process. Another possibility could be to execute {\f1\fs20 DB.Synchronous := smOff} and/or {\f1\fs20 DB.LockingMode := lmExclusive} at {\i @*SQLite3@} engine level before process: in case of power loss at wrong time it may corrupt the database file, but it will increase the rate by a factor of 50 (with hard drive), as stated by the "{\i off}" and "{\i off exc}" rows of the table - see @60@. Note that by default, the {\i @*FireDAC@} library set both options, so results above are to be compared with "{\i SQLite3 off exc}" rows. In {\i SQLite3} direct mode, BATCH process benefits of multi-INSERT statements (just llike external databases): it explain why {\f1\fs20 BatchAdd()} is faster than plain {\f1\fs20 Add()}, even in the slowest and safest "{\i file full}" mode.
For our direct {\i @*Oracle@} access {\f1\fs20 SynDBOracle.pas} unit, and for {\f1\fs20 SynDBZeos.pas} or {\f1\fs20 SynDBFireDAC.pas} (known as {\i Array DML} in {\i FireDAC/AnyDAC}) libraries, BATCH process benefits of the @*array bind@ing feature a lot.
For most engines, our ORM kernel is able to generate the appropriate SQL statement for speeding up bulk insertion. For instance:
- {\i SQlite3, @*MySQL@, @*PostgreSQL@, MSSQL 2008, @*DB2@, @*MySQL@} or {\i @*NexusDB@} handle {\f1\fs20 INSERT} statements with multiple {\f1\fs20 INSERT INTO .. VALUES (..),(..),(..)..};
- {\i Oracle} handles {\f1\fs20 INSERT INTO .. INTO .. SELECT 1 FROM DUAL} (weird syntax, isn't it?);
- {\i @*Firebird@} implements {\f1\fs20 EXECUTE BLOCK}.
As a result, some engines show a nice speed boost when {\f1\fs20 BatchAdd()} is used. Even {\i SQLite3} is faster when used as external engine, in respect to direct execution! This feature is at ORM/SQL level, so it benefits to any external database library. Of course, if a given library has a better implementation pattern (e.g. our direct {\i Oracle}, {\i Zeos} or {\i FireDAC} with native array binding), it is used instead.
{\i @*MongoDB@} bulk insertion has been implemented, which shows an amazing speed increase in Batch mode. Depending on the {\i MongoDB} {\i write concern} mode, insertion speed can be very high: by default, every write process will be acknowledge by the server, but you can by-pass this request if you set the {\f1\fs20 wcUnacknowledged} mode - note that in this case, any error (e.g. an unique field duplicated value) will never be notified, so it should not be used in production, unless you need this feature to quickly populate a database, or consolidate some data as fast as possible.
:   Reading speed
Now the same data is retrieved via the ORM layer:
- 'By one' states that one object is read per call (ORM generates a {\f1\fs20 SELECT * FROM table WHERE ID=?} for {\f1\fs20 Client.Retrieve()} method);
- 'All *' is when all 5000 objects are read in a single call (i.e. running {\f1\fs20 SELECT * FROM table} from a {\f1\fs20 FillPrepare()} method call), either forced to use the virtual table layer, or with direct static call.
Here are some reading speed values, in objects/second:
|%30%15%15%15
||\b By one|All Virtual|All Direct\b0
|{\b SQLite3 (file full)}|127284|558721|550842
|{\b SQLite3 (file off)}|126896|549450|526149
|{\b SQLite3 (file off exc)}|128077|557537|535905
|{\b SQLite3 (mem)}|127106|557537|563316
|{\b TObjectList (static)}|300012|912408|913742
|{\b TObjectList (virtual)}|303287|402706|866551
|{\b SQLite3 (ext full)}|135380|267436|553158
|{\b SQLite3 (ext off)}|133696|262977|543065
|{\b SQLite3 (ext off exc)}|134698|264186|558596
|{\b SQLite3 (ext mem)}|137487|259713|557475
|{\b WinHTTP SQLite3}|2198|209231|340460
|{\b Sockets SQLite3}|8524|210260|387687
|{\b MongoDB (ack)}|8002|262353|271268
|{\b MongoDB (no ack)}|8234|272079|274582
|{\b ODBC SQLite3}|19461|136600|201280
|{\b ZEOS SQlite3}|33541|200835|306955
|{\b FireDAC SQlite3}|7683|83532|112470
|{\b UniDAC SQlite3}|2522|74030|96420
|{\b ODBC Firebird}|3446|69607|97585
|{\b ZEOS Firebird}|20296|114676|117210
|{\b FireDAC Firebird}|2376|46276|56269
|{\b UniDAC Firebird}|2189|66886|88102
|{\b Jet}|2640|166112|258277
|{\b NexusDB}|1413|120845|208246
|{\b Oracle}|1558|120977|159861
|{\b ZEOS Oracle}|1420|110367|137982
|{\b ODBC Oracle}|1620|43441|45764
|{\b FireDAC Oracle}|1231|42149|54795
|{\b UniDAC Oracle}|688|27083|30093
|{\b BDE Oracle}|860|3870|4036
|{\b MSSQL local}|10135|210837|437905
|{\b ODBC MSSQL}|12458|147544|256502
|{\b FireDAC MSSQL}|3776|72123|94091
|{\b UniDAC MSSQL}|2505|93231|135932
|{\b ODBC DB2}|7649|84880|124486
|{\b FireDAC DB2}|3155|71456|88264
|{\b ZEOS PostgreSQL}|8833|158760|223583
|{\b ODBC PostgreSQL}|10361|85680|120913
|{\b FireDAC PostgreSQL}|2261|58252|79002
|{\b UniDAC PostgreSQL}|864|86900|122856
|{\b ODBC MySQL}|10143|65538|82447
|{\b ZEOS MySQL}|2052|171803|245772
|{\b FireDAC MySQL}|3636|75081|105028
|{\b UniDAC MySQL}|4798|99940|146968
|%
The {\i @*SQLite3@} layer gives amazing reading results, which makes it a perfect fit for most typical ORM use. When running with {\f1\fs20 DB.LockingMode := lmExclusive} defined (i.e. "off exc" rows), reading speed is very high, and benefits from exclusive access to the database file - see @60@. External database access is only required when data is expected to be shared with other processes, or for better scaling: e.g. for physical n-Tier installation with dedicated database server(s).
In the above table, it appears that all libraries based on {\f1\fs20 DB.pas} are slower than the others for reading speed. In fact, {\f1\fs20 TDataSet} sounds to be a real bottleneck, due to its internal data marshalling. Even {\i @*FireDAC@}, which is known to be very optimized for speed, is limited by the {\f1\fs20 TDataSet} structure. Our direct classes, or even ZEOS/ZDBC performs better, since they are able to output JSON content with no additional marshalling, via a dedicated {\f1\fs20 ColumnsToJSON()} method.
For both writing and reading, {\f1\fs20 TObjectList} / {\f1\fs20 TSQLRestStorageInMemory} engine gives impressive results, but has the weakness of being in-memory, so it is not ACID by design, and the data has to fit in memory. Note that indexes are available for IDs and {\f1\fs20 stored @*AS_UNIQUE@} properties.
As a consequence, search of non-unique values may be slow: the engine has to loop through all rows of data. But for unique values (defined as {\f1\fs20 stored AS_UNIQUE}), both insertion and search speed is awesome, due to its optimized O(1) hash algorithm - see the following benchmark, especially the "{\i By name}" row for "{\i TObjectList}" columns, which correspond to a search of an unique {\f1\fs20 RawUTF8} property value via this hashing method.
|%9%10%10%10%10%10%10%10%10%9%9
| |\b SQLite3 (file full)|SQLite3 (file off)|SQLite3 (mem)|TObjectList (static) |TObjectList (virt.)|SQLite3 (ext file full)|SQLite3 (ext file off)|SQLite3 (ext mem)|Oracle|Jet\b0
|{\b By one}|10461|10549|44737|103577|103553|43367|44099|45220|901|1074
|{\b By name}|9694|9651|32350|70534|60153|22785|22240|23055|889|1071
|{\b All Virt.}|167095|162956|168651|253292|118203|97083|90592|94688|56639|52764
|{\b All Direct}|167123|144250|168577|254284|256383|170794|165601|168856|88342|75999
|%
Above table results were run on a Core 2 duo laptop, so numbers are lower than with the previous tables.
During the tests, internal caching - see @37@ and @38@ - was disabled, so you may expect speed enhancements for real applications, when data is more read than written: for instance, when an object is retrieved from the cache, you achieve more than 1,00,000 read requests per second, whatever database is used.
:   Analysis and use case proposal
When declared as virtual table (via a {\f1\fs20 VirtualTableRegister} call), you have the full power of SQL (including @*JOIN@s) at hand, with incredibly fast @*CRUD@ operations: 100,000 requests per second for objects read and write, including serialization and Client-Server communication!
Some providers are first-class citizens to {\i mORMot}, like {\i @*SQLite3@}, {\i @*Oracle@}, {\i @*MS SQL@}, @*PostgreSQL@, @*MySQL@ or IBM @*DB2@. You can connect to them without the bottleneck of the {\f1\fs20 DB.pas} unit, nor any restriction of your {\i Delphi} license (a {\i Starter edition} is enough).
First of all, {\i SQLite3} is still to be considered, even for a production server. Thanks to {\i mORMot}'s architecture and design, this "embedded" database could be used as main database engine for a client-server application with heavy concurrent access - if you have doubts about its scaling abilities, see @25@. Here, "embedded" is not restricted to "mobile", but sounds like a self-contained, zero-configuration proven engine.
The remote access via HTTP gives pretty good results, and in this local benchmark, plain socket client (i.e. {\f1\fs20 TSQLDBSocketConnectionProperties} class) gives better results that the {\f1\fs20 WinHTTP} API (using {\f1\fs20 TSQLDBWinHTTPConnectionProperties} on the client side). But in real use, e.g. over the Internet, the {\f1\fs20 WinHTTP} API has been reported as more stable, so may be preferred on production. With a {\i SQlite3} backend, this offers pretty good performance, and the benefit of using standard HTTP for its transport.
Most recognized {\i closed source} databases are available:
- Direct access to {\i Oracle} gives impressive results in BATCH mode (aka @*array bind@ing). It may be an obligation if your end-customer stores already its data in such a server, for instance, and want to leverage the licensing cost of its own IT solution. {\i Oracle Express} edition is free, but somewhat heavy and limited in terms of data/hardware size (see its licensing terms);
- {\i MS SQL Server}, directly accessed via {\i OleDB} (or {\i ODBC}) gives pretty good timing. A {\i MS SQL Server 2008 R2 Express} instance is pretty well integrated with the {\i Windows} environment, for a very affordable price (i.e. for free) - the {\i LocalDB} (MSI installer) edition is enough to start with, but also with data/hardware size limitation, just like {\i Oracle Express};
- IBM {\i DB2} is another good candidate, and the {\i Express-C} ("C" standing for Community) offers a no-charge opportunity to run an industry standard engine, with no restriction on the data size, and somewhat high hardware limitations (16 GB of RAM and 2 CPU cores for the latest 10.5 release) or enterprise-level features;
- We did not include {\i Informix} numbers here, since support for this database was provided by an user patch - thanks Esteban Martin for sharing! - and we do not have any such server available here;
- {\i @*NexusDB@} may be considered, if you have existing {\i Delphi} code and data - but it is less known and recognized as the its commercial competitors.
{\i Open Source} databases are worth considering, especially in conjunction with an Open Source framework like {\i mORMot}:
- {\i MySQL} is the well-known engine used by a lot of web sites, mainly with {\i LAMP} ({\i @*Linux@ Apache MySQL PHP}) configurations. Windows is not the best platform to run it, but it could be a fairly good candidate, especially in its {\i MariaDB} fork, which sounds more attractive those days than the official main version, owned by Oracle;
- {\i PostgreSQL} is an Enterprise class database, with amazing features among its Open Source alternatives, and really competes with commercial solutions. Even under Windows, we think it is easy to install and administrate, and uses less resource than the other commercial engines.
- {\i @*Firebird@} gave pretty consistent timing, when accessed via Zeos/ZDBC. We show here the embedded version, but the server edition is worth considering, since a lot of {\i Delphi} programmers are skilled with this free alternative to {\i Interbase};
- {\i @*MongoDB@} appears as a serious competitor to SQL databases, with the potential benefit of horizontal scaling and installation/administration ease - performance is very high, and its document-based storage fits perfectly with {\i mORMot}'s advanced ORM features like @29@.
To access those databases, @*OleDB@, @*ODBC@ or @*ZDBC@ providers may also be used, with direct access. {\i mORMot} is a very open-minded rodent: you can use any {\f1\fs20 DB.pas} provider, e.g. {\i @*FireDAC@}, {\i @*UniDAC@}, {\i @*DBExpress@}, {\i @*NexusDB@} or even the {\i @*BDE@}, but with the additional layer introduced by using a {\f1\fs20 @*TDataSet@} instance, at reading.
Therefore, the typical use may be the following:
|%27%73
|\b Database|Use case\b0
|internal @*SQLite3@ file|Created by default.\line General safe data handling, with amazing speed in "off exc" mode
|internal SQLite3 in-memory|Created with {\f1\fs20 ':memory:'} file name.\line Fast data handling with no persistence (e.g. for testing or temporary storage)
|{\f1\fs20 TObjectList} static|Created with {\f1\fs20 StaticDataCreate}.\line Best possible performance for small amount of data, without ACID nor SQL
|{\f1\fs20 TObjectList} virtual|Created with {\f1\fs20 VirtualTableRegister}.\line Best possible performance for SQL over small amount of data (or even unlimited amount under @*Win64@), if ACID is not required nor complex SQL
|external SQLite3 file|Created with {\f1\fs20 @*VirtualTableExternalRegister@}\line External back-end, e.g. for disk spanning
|external SQLite3 in-memory|Created with {\f1\fs20 VirtualTableExternalRegister} and {\f1\fs20 ':memory:'}\line Fast external back-end (e.g. for testing)
|external @*Oracle@ / @*MS SQL@ / @*DB2@ / @*PostgreSQL@ / @*MySQL@ / @*Informix@ / @*Firebird@|Created with {\f1\fs20 VirtualTableExternalRegister}\line Fast, secure and industry standard back-ends; data can be shared outside {\i mORMot}
|external @*NexusDB@|Created with {\f1\fs20 VirtualTableExternalRegister}\line The free embedded version let the whole engine be included within your executable, and use any existing code, but {\i SQlite3} sounds like a better option
|external @*Jet/MSAccess@|Created with {\f1\fs20 VirtualTableExternalRegister}\line Could be used as a data exchange format (e.g. with Office applications)
|external Zeos|Created with {\f1\fs20 VirtualTableExternalRegister}\line Allow access to several external engines, with direct @*Zeos@/@*ZDBC@ access which will by-pass the {\f1\fs20 DB.pas} unit and its {\f1\fs20 TDataSet} bottleneck - and we will also prefer an active Open Source project!
|external @*FireDAC@/@*UniDAC@|Created with {\f1\fs20 VirtualTableExternalRegister}\line Allow access to several external engines, including the {\f1\fs20 DB.pas} unit and its {\f1\fs20 TDataSet} bottleneck
|external {\i MongoDB}|Created with {\f1\fs20 @*StaticMongoDBRegister@()}\line High-speed document-based storage, with horizontal scaling and advanced query abilities of nested sub-documents
|%
Whatever database back-end is used, don't forget that {\i mORMot} design will allow you to switch from one library to another, just by changing a {\f1\fs20 TSQLDBConnectionProperties} class type. And note that you can {\i mix} external engines, on purpose: you are not tied to one single engine, but the database access can be tuned for each ORM table, according to your project needs.
\page
: SQLite3 implementation
Beginning with the revision 1.15 of the framework, the {\i @**SQLite3@} engine itself has been separated from our {\f1\fs20 mORMotSQLite3.pas} unit, and defined as a stand-alone unit named {\f1\fs20 SynSQLite3.pas}. See @SDD-DI-2.2.1@.
It can be used therefore:
- Either stand-alone with direct access of all its features, even using its lowest-level C API, via {\f1\fs20 SynSQLite3.pas} - but you won't be able to switch to another database engine easily;
- Or stand-alone with high-level SQL access, using our {\f1\fs20 @*SynDB@.pas} generic access classes, via {\f1\fs20 SynDBSQLite3.pas} - so you will be able to change to any other database engine (e.g. @*MS SQL@, @*PostgreSQL@, @*MySQL@ or @*Oracle@) when needed;
- Or Client-Server based access with all our @*ORM@ features - see {\f1\fs20 mORMotSQLite3.pas}.
We'll define here some highlights specific to our own implementation of the {\i SQLite3} engine, and let you consult the official documentation of this great Open Source project at @http://sqlite.org for general information about its common features.
:  Statically linked or using external dll
Since revision 1.18 of the framework, our {\f1\fs20 SynSQlite3.pas} unit is able to access the {\i SQLite3} engine in two ways:
- Either {\i statically linked} within the project executable;
- Or from an external {\f1\fs20 sqlite3.dll} library file.
The {\i SQLite3} APIs and constants are defined in {\f1\fs20 SynSQlite3.pas}, and accessible via a {\f1\fs20 TSQLite3Library} class definition. It defines a global {\f1\fs20 sqlite3} variable as such:
!var
!  sqlite3: TSQLite3Library;
To use the {\f1\fs20 SQLite3} engine, an instance of {\f1\fs20 TSQLite3Library} class shall be assigned to this global variable. Then all {\i mORMot}'s calls will be made through it, calling e.g. {\f1\fs20 sqlite3.open()} instead of {\f1\fs20 sqlite3_open()}.
There are two implementation classes:
\graph HierTSQLite3Library TSQLite3Library classes hierarchy
\TSQLite3LibraryStatic\TSQLite3Library
\TSQLite3LibraryDynamic\TSQLite3Library
\
|%32%32%36
|\b Class|Unit|Purpose\b0
|{\f1\fs20 TSQLite3LibraryStatic}|{\f1\fs20 SynSQLite3Static.pas}|Statically linked engine ({\f1\fs20 sqlite3.obj}/{\f1\fs20 sqlite3.o} within the {\f1\fs20 .exe})
|{\f1\fs20 TSQLite3LibraryDynamic}|{\f1\fs20 SynSQLite3.pas}|Instantiate an external {\f1\fs20 sqlite3.dll} instance
|%
Referring to {\f1\fs20 SynSQLite3Static.pas} in the {\f1\fs20 uses} clause of your project is enough to link the {\f1\fs20 .obj/.o} engine into your executable.
{\i Warning - breaking change}: before version 1.18 of the framework, link of static {\f1\fs20 .obj} was forced - so you must now add a reference to {\f1\fs20 SynSQLite3Static} in your project {\f1\fs20 uses} clause to work as expected.
In order to use an external {\f1\fs20 sqlite3.dll} library, you have to set the global {\f1\fs20 sqlite3} variable as such:
! FreeAndNil(sqlite3); // release any previous instance (e.g. static)
!! sqlite3 := TSQLite3LibraryDynamic.Create;
Of course, {\f1\fs20 FreeAndNil(sqlite3)} is not mandatory, and should be necessary only to avoid any memory leak if another {\i SQLite3} engine instance was allocated (may be the case if {\f1\fs20 SynSQLite3Static} is referred somewhere in your project's units).
Here are some benchmarks, compiled with {\i Delphi XE3}, run in a 32-bit project, using either the static {\f1\fs20 bcc}-compiled engine, or an external {\f1\fs20 sqlite3.dll}, compiled via {\f1\fs20 MinGW} or Visual C++.
:   Static bcc-compiled .obj
First of all, our version included with {\f1\fs20 SynSQLite3Static.pas} unit, is to be benchmarked.
\line {\i Writing speed}
|%30%15%15%15%15
||{\b Direct}|{\b Batch}|{\b Trans}|{\b Batch Trans}
|{\b SQLite3 (file full)}|477|389|97633|122865
|{\b SQLite3 (file off)}|868|869|96827|125862
|{\b SQLite3 (mem)}|84642|108624|104947|135105
|{\b TObjectList (static)}|338478|575373|337336|572147
|{\b TObjectList (virtual)}|338180|554446|331873|575837
|{\b SQLite3 (ext full)}|486|496|101419|7011
|{\b SQLite3 (ext off)}|799|303|105402|135109
|{\b SQLite3 (ext mem)}|93893|129550|109027|152811
|%
\line {\i Reading speed}
|%30%15%15%15
||{\b By one}|{\b All Virtual}|{\b All Direct}
|{\b SQLite3 (file full)}|26924|494559|500200
|{\b SQLite3 (file off)}|27750|496919|502714
|{\b SQLite3 (mem)}|124402|444404|495392
|{\b TObjectList (static)}|332778|907605|910249
|{\b TObjectList (virtual)}|331038|404891|905961
|{\b SQLite3 (ext full)}|102707|261547|521322
|{\b SQLite3 (ext off)}|131130|255806|513505
|{\b SQLite3 (ext mem)}|135784|248780|502664
|%
Good old {\i Borland C++ builder} produces some efficient code here. Those numbers are very good, when compared to the other two options. Probably, using {\i @*FastMM4@} as memory manager and tuned compilation options does make sense.
:   Official MinGW-compiled sqlite3.dll
Here we used the official {\f1\fs20 sqlite3.dll} library, as published in the @http://sqlite.org web site, and compiled with the MinGW/GCC compiler.
\line {\i Writing speed}
|%30%15%15%15%15
||{\b Direct}|{\b Batch}|{\b Trans}|{\b Batch Trans}
|{\b SQLite3 (file full)}|418|503|86322|119420
|{\b SQLite3 (file off)}|918|873|93196|127317
|{\b SQLite3 (mem)}|83108|106951|99892|138003
|{\b TObjectList (static)}|320204|573723|324696|547465
|{\b TObjectList (virtual)}|323247|563697|324443|564716
|{\b SQLite3 (ext full)}|501|410|100152|133679
|{\b SQLite3 (ext off)}|913|438|102806|135545
|{\b SQLite3 (ext mem)}|96028|122798|108363|150920
|%
\line {\i Reading speed}
|%30%15%15%15
||{\b By one}|{\b All Virtual}|{\b All Direct}
|{\b SQLite3 (file full)}|26883|473529|438904
|{\b SQLite3 (file off)}|27729|472188|451304
|{\b SQLite3 (mem)}|116550|459432|457959
|{\b TObjectList (static)}|318248|891265|905469
|{\b TObjectList (virtual)}|327739|359040|892697
|{\b SQLite3 (ext full)}|127346|180812|370288
|{\b SQLite3 (ext off)}|127749|227759|438096
|{\b SQLite3 (ext mem)}|129792|224386|436338
|%
:   Visual C++ compiled sqlite3.dll
The {\i Open Source wxsqlite} project provides a {\f1\fs20 sqlite3.dll} library, compiled with {\i Visual C++}, and including RC4 and @*AES@ 128/256 encryption (better than the basic encryption implemented in {\f1\fs20 SynSQLite3Static.pas}) - not available in the official library.
See @http://sourceforge.net/projects/wxcode/files/Components/wxSQLite3 to download the corresponding source code, and compiled {\f1\fs20 .dll}.
\line {\i Writing speed}
|%30%15%15%15%15
||{\b Direct}|{\b Batch}|{\b Trans}|{\b Batch Trans}
|{\b SQLite3 (file full)}|470|498|93801|112170
|{\b SQLite3 (file off)}|886|819|90298|132883
|{\b SQLite3 (mem)}|86897|110287|105207|140896
|{\b TObjectList (static)}|332005|596445|321357|570776
|{\b TObjectList (virtual)}|327225|585000|329272|579240
|{\b SQLite3 (ext full)}|459|503|91086|140599
|{\b SQLite3 (ext off)}|501|519|110338|150394
|{\b SQLite3 (ext mem)}|98112|133276|117346|158634
|%
\line {\i Reading speed}
|%30%15%15%15
||{\b By one}|{\b All Virtual}|{\b All Direct}
|{\b SQLite3 (file full)}|28527|516689|521159
|{\b SQLite3 (file off)}|28927|513769|519156
|{\b SQLite3 (mem)}|127740|529100|523176
|{\b TObjectList (static)}|335053|869262|879352
|{\b TObjectList (virtual)}|334739|410374|885269
|{\b SQLite3 (ext full)}|132594|258371|506277
|{\b SQLite3 (ext off)}|138159|260892|507717
|{\b SQLite3 (ext mem)}|139567|254919|516208
|%
Under {\i Windows}, the {\i Visual C++} compiler gives very good results. It is a bit faster than the other two, despite a somewhat less efficient virtual table process.
As a conclusion, our {\f1\fs20 SynSQLite3Static.pas} statically linked implementation sounds like the best overall approach for Windows 32-bit: best speed for virtual tables (which is the core of our ORM), and no {\i dll hell}. No library to deploy and copy, everything is embedded in the project executable, ready to run as expected. External {\f1\fs20 sqlite3.dll} will be used for cross-platform support, and when targeting @*64-bit@ Windows applications.
:14  Prepared statement
In order to speed up the time spent in the {\i SQLite3} engine (it may be useful for high-end servers), the framework is able to natively handle @*prepared@ @*SQL@ statements.
Starting with version 1.12 of the framework, we added an internal SQL statement @*cache@ in the database access, available for all SQL request. Previously, only the one-record SQL {\f1\fs20 SELECT * FROM ... WHERE RowID=...} was prepared (used e.g. for the {\f1\fs20 @*TSQLRest@. Retrieve} method).
That is, if a previous SQL statement is run with some given parameters, a prepared version, available in cache, is used, and new parameters are bounded to it before the execution by {\i SQLite3}.
In some cases, it can speed the {\i SQLite3} process a lot. From our profiling, prepared statements make common requests (i.e. select / insert / update on one row) at least two times faster, on an in-memory database ({\f1\fs20 ':memory:'} specified as file name).
In order to use this statement caching, any SQL statements must have the parameters to be surrounded with '{\f1\fs20 :(}' and '{\f1\fs20 ):}'. The SQL format was indeed enhanced by adding an optional way of marking parameters inside the SQL request, to enforce statement preparing and caching.
Therefore, there are now two ways of writing the same SQL request:
Write the SQL statement as usual:
$SELECT * FROM TABLE WHERE ID=10;
in this case, the SQL will be parsed by the {\i SQLite3} engine, a statement will be compiled, then run.
Use the new optional markers to identify the changing parameter:
$SELECT * FROM TABLE WHERE ID=:(10):;
in this case, any matching already prepared statement will be re-used for direct run.
In the later case, an internal pool of prepared {\f1\fs20 TSQLRequest} statements is maintained. The generic SQL code used for the matching will be this one:
$SELECT * FROM TABLE WHERE ID=?;
and the integer value 10 will be bounded to the prepared statement before execution.
Example of possible inlined values are (note double " @*quotes@ are allowed for the text parameters, whereas SQL statement should only use single ' quotes):
$:(1234): :(12.34): :(12E-34): :("text"): :('It''s great'):
All internal SQL statement generated by the @*ORM@ are now using this new parameter syntax.
For instance, here is how an object deletion is implemented for the {\i SQlite3} engine:
!function TSQLRestServerDB.EngineDelete(Table: TSQLRecordClass; ID: TID): boolean;
!begin
!  if Assigned(OnUpdateEvent) then
!    OnUpdateEvent(self,seDelete,Table,ID); // notify BEFORE deletion
!  result := ExecuteFmt('DELETE FROM % WHERE RowID=:(%):;',[Table.SQLTableName,ID]);
!end;
Using {\f1\fs20 :(%):} will let the {\f1\fs20 DELETE FROM table_name WHERE RowID=?} statement be prepared and reused between calls.
In your code, you should better use, for instance:
! aName := OneFieldValue(TSQLMyRecord,'Name','ID=:(%):',[aID]);
 or even easier
! aName := OneFieldValue(TSQLMyRecord,'Name','ID=?',[],[aID]);
 instead of
! aName := OneFieldValue(TSQLMyRecord,'Name','ID=%',[aID]);
 or instead of a plain
! aName := OneFieldValue(TSQLMyRecord,'Name','ID='+Int32ToUtf8(aID));
In fact, from your client code, you may not use directly the {\f1\fs20 :(...):} expression in your request, but will rather use the overloaded {\f1\fs20 TSQLRecord.Create, TSQLRecord.FillPrepare, TSQLRecord.CreateAndFillPrepare, TSQLRest.OneFieldValue, TSQLRest.MultiFieldValues, TQLRestClient.ExecuteFmt} and {\f1\fs20 TSQLRestClient.ListFmt} methods, available since revision 1.15 of the framework, which will accept both '%' and '?' characters in the SQL WHERE format text, in-lining '?' parameters with proper {\f1\fs20 :(...):} encoding and quoting the {\f1\fs20 @*RawUTF8@} / strings parameters on purpose.
I found out that this SQL format enhancement is much easier to use (and faster) in the {\i Delphi} code than using parameters by name or by index, like in this classic VCL code:
$SQL.Text := 'SELECT Name FROM Table WHERE ID=:Index';
$SQL.ParamByName('Index').AsInteger := aID;
At a lowest-level, in-lining the bounds values inside the statement enabled better serialization in a Client-Server architecture, and made caching easier on the Server side: the whole SQL query contains all parameters within one unique {\f1\fs20 RawUTF8} value, and can be therefore directly compared to the cached entries. As such, our framework is able to handle prepared statements without keeping bound parameters separated from the main SQL text.
It is also worth noting that external databases (see next paragraph) will also benefit from this statement preparation. Inlined values will be bound separately to the external SQL statement, to achieve the best speed possible.
:  R-Tree inclusion
Since the 2010-06-25 source code repository update, the @*RTREE@ extension is now compiled by default within all supplied {\f1\fs20 .obj} files.
An R-Tree is a special index that is designed for doing range queries. R-Trees are most commonly used in geospatial systems where each entry is a rectangle with minimum and maximum X and Y coordinates. Given a query rectangle, an R-Tree is able to quickly find all entries that are contained within the query rectangle or which overlap the query rectangle. This idea is easily extended to three dimensions for use in CAD systems. R-Trees also find use in time-domain range look-ups. For example, suppose a database records the starting and ending times for a large number of events. A R-Tree is able to quickly find all events, for example, that were active at any time during a given time interval, or all events that started during a particular time interval, or all events that both started and ended within a given time interval. And so forth. See @http://www.sqlite.org/rtree.html
A dedicated @*ORM@ class, named {\f1\fs20 TSQLRecordRTree}, is available to create such tables. It inherits from {\f1\fs20 TSQLRecordVirtual}, like the other @*virtual table@s types (e.g. {\f1\fs20 TSQLRecordFTS5}).
Any record which inherits from this {\f1\fs20 TSQLRecordRTree} class must have only {\f1\fs20 sftFloat} (i.e. {\i Delphi} {\f1\fs20 @*double@}) @*published properties@ grouped by pairs, each as minimum- and maximum-value, up to 5 dimensions (i.e. 11 columns, including the ID property). Its {\f1\fs20 ID: @*TID@} property must be set before adding a {\f1\fs20 TSQLRecordRTree} to the database, e.g. to link an R-Tree representation to a regular {\f1\fs20 @*TSQLRecord@} table containing the main data.
Queries against the ID or the coordinate ranges are almost immediate: so you can e.g. extract some coordinates box from the main regular {\f1\fs20 TSQLRecord} table, then use a {\f1\fs20 TSQLRecordRTree}-joined query to make the process faster; this is exactly what the {\f1\fs20 TSQLRestClient. RTreeMatch} method offers: for instance, running with {\f1\fs20 aMapData. @*Blob@Field} filled with {\f1\fs20 [-81,-79.6,35,36.2]} the following lines:
! aClient.RTreeMatch(TSQLRecordMapData,'BlobField',TSQLRecordMapBox,
!   aMapData.BlobField,ResultID);
will execute the following @*SQL@ statement:
$ SELECT MapData.ID From MapData, MapBox WHERE MapData.ID=MapBox.ID
$  AND minX>=:(-81.0): AND maxX<=:(-79.6): AND minY>=:(35.0): AND :(maxY<=36.2):
$  AND MapBox_in(MapData.BlobField,:('\uFFF0base64encoded-81,-79.6,35,36.2'):);
The {\f1\fs20 MapBox_in} @*SQL function@ is registered in {\f1\fs20 @*TSQLRestServerDB@. Create} constructor for all {\f1\fs20 TSQLRecordRTree} classes of the current database model. Both {\f1\fs20 BlobToCoord} and {\f1\fs20 ContainedIn} class methods are used to handle the box storage in the BLOB. By default, it will process a raw {\f1\fs20 array of double}, with a default box match (that is {\f1\fs20 ContainedIn} method will match the simple {\f1\fs20 minX>=...maxY<=...} where clause).
:8  FTS3/FTS4/FTS5
@**FTS@3/FTS4/FTS5 are {\i @*SQLite3@} @*virtual table@ modules that allow users to perform full-text searches on a set of documents. The most common (and effective) way to describe full-text searches is "what Google, Yahoo and Altavista do with documents placed on the World Wide Web". Users input a term, or series of terms, perhaps connected by a binary operator or grouped together into a phrase, and the full-text query system finds the set of documents that best matches those terms considering the operators and groupings the user has specified.
See @http://www.sqlite.org/fts3.html as reference material about FTS3/FTS4 usage in {\i SQLite3}, and @https://www.sqlite.org/fts5.html about FTS5. In short, FTS5 is a new version of FTS4 that includes various fixes and solutions for problems that could not be fixed in FTS4 without sacrificing backwards compatibility.
Since recent versions of the framework, the {\f1\fs20 sqlite3.obj/.o} static file available with the distribution includes the FTS3/FTS4/FTS5 engines (also on other platforms with {\f1\fs20 FPC}).
:   Dedicated FTS3/FTS4/FTS5 record type
In order to allow easy use of the @*FTS@ feature, some types have been defined:
- {\f1\fs20 TSQLRecordFTS3} to create a FTS3 table with default "simple" stemming;
- {\f1\fs20 TSQLRecordFTS3Porter} to create a FTS3 table using the {\i Porter Stemming} algorithm (see below);
- {\f1\fs20 TSQLRecordFTS3Unicode61} to create a FTS3 table using the {\i Unicode61 Stemming} algorithm (see below);
- {\f1\fs20 TSQLRecordFTS4} to create a FTS4 table with default "simple" stemming;
- {\f1\fs20 TSQLRecordFTS4Porter} to create a FTS4 table using the {\i Porter Stemming} algorithm;
- {\f1\fs20 TSQLRecordFTS4Unicode61} to create a FTS4 table using the {\i Unicode61 Stemming};
- {\f1\fs20 TSQLRecordFTS5} to create a FTS5 table with default "simple" stemming;
- {\f1\fs20 TSQLRecordFTS5Porter} to create a FTS5 table using the {\i Porter Stemming} algorithm;
- {\f1\fs20 TSQLRecordFTS5Unicode61} to create a FTS5 table using the {\i Unicode61 Stemming};
The following graph will detail this class hierarchy:
\graph FTSORMClasses FTS ORM classes
\TSQLRecord\TSQLRecordVirtual
\TSQLRecordVirtual\TSQLRecordFTS3
\TSQLRecordFTS3\TSQLRecordFTS3Porter
\TSQLRecordFTS3\TSQLRecordFTS4
\TSQLRecordFTS3\TSQLRecordFTS3Unicode61
\TSQLRecordFTS4\TSQLRecordFTS4Porter
\TSQLRecordFTS4\TSQLRecordFTS5
\TSQLRecordFTS4\TSQLRecordFTS4Unicode61
\TSQLRecordFTS5\TSQLRecordFTS5Porter
\TSQLRecordFTS5\TSQLRecordFTS5Unicode61
\
In practice, you should use {\f1\fs20 TSQLRecordFTS5} when working with latin content, {\f1\fs20 TSQLRecordFTS5Unicode61} for better non-latin support, and {\f1\fs20 TSQLRecordFTS5Porter} if your content is some plain English text.
:   Stemming
The "stemming" algorithm - see @http://sqlite.org/fts3.html#tokenizer - is the way the english text is parsed for creating the word index from raw text.
The {\i simple} (default) tokenizer extracts tokens from a document or basic @*FTS@ full-text query according to the following rules:
- A term is a contiguous sequence of eligible characters, where eligible characters are all alphanumeric characters, the "_" character, and all characters with UTF code-points greater than or equal to 128. All other characters are discarded when splitting a document into terms. Their only contribution is to separate adjacent terms.
- All uppercase characters within the ASCII range (UTF code-points less than 128), are transformed to their lowercase equivalents as part of the tokenization process. Thus, full-text queries are case-insensitive when using the simple tokenizer.
For example, when a document containing the text "{\i Right now, they're very frustrated.}", the terms extracted from the document and added to the full-text index are, in order, "{\f1\fs20 right now they re very frustrated}". Such a document will match a full-text query such as "{\f1\fs20 MATCH 'Frustrated'}", as the simple tokenizer transforms the term in the query to lowercase before searching the full-text index.
The {\i Porter Stemming algorithm} tokenizer uses the same rules to separate the input document into terms, but as well as folding all terms to lower case it uses the {\i Porter Stemming} algorithm to reduce related English language words to a common root. For example, using the same input document as in the paragraph above, the porter tokenizer extracts the following tokens: "{\f1\fs20 right now thei veri frustrat}". Even though some of these terms are not even English words, in some cases using them to build the full-text index is more useful than the more intelligible output produced by the simple tokenizer. Using the porter tokenizer, the document not only matches full-text queries such as "{\f1\fs20 MATCH 'Frustrated'}", but also queries such as "{\f1\fs20 MATCH 'Frustration'}", as the term "{\i Frustration}" is reduced by the Porter stemmer algorithm to "{\i frustrat}" - just as "{\i Frustrated}" is. So, when using the porter tokenizer, FTS is able to find not just exact matches for queried terms, but matches against similar English language terms. For more information on the Porter Stemmer algorithm, please refer to the @http://tartarus.org/~martin/PorterStemmer page.
The {\i Unicode61 Stemming algorithm} tokenizer works very much like "simple" except that it does simple unicode case folding according to rules in Unicode Version 6.1 and it recognizes unicode space and punctuation characters and uses those to separate tokens. By default, "Unicode61" also removes all diacritics from Latin script characters.
:24   FTS searches
A good approach is to store your data in a regular {\f1\fs20 @*TSQLRecord@} table, then store your text content in a separated @*FTS@ table, associated to this {\f1\fs20 TSQLRecordFTS5} table via its {\f1\fs20 ID / DocID} property. Note that for {\f1\fs20 TSQLRecordFTS*} types, the {\f1\fs20 ID} property was renamed as {\f1\fs20 DocID}, which is the internal name for the FTS virtual table definition of its unique integer key {\f1\fs20 ID} property.
For example (extracted from the regression @*test@ code), you can define this new class:
!  TSQLFTSTest = class(TSQLRecordFTS5)
!  private
!    fSubject: RawUTF8;
!    fBody: RawUTF8;
!  published
!    property Subject: RawUTF8 read fSubject write fSubject;
!    property Body: RawUTF8 read fBody write fBody;
!  end;
Note that FTS tables must only content @*UTF-8@ text field, that is {\f1\fs20 @*RawUTF8@} (under {\i Delphi} 2009 and up, you could also use the Unicode {\f1\fs20 string} type, which is mapped as a UTF-8 text field for the {\i SQLite3} engine).
Then you can add some {\i Body/Subject} content to this FTS table, just like any regular {\f1\fs20 TSQLRecord} content, via the @*ORM@ feature of the framework:
!  FTS := TSQLFTSTest.Create;
!  try
!    Check(aClient.TransactionBegin(TSQLFTSTest)); // MUCH faster with this
!    for i := StartID to StartID+COUNT-1 do
!    begin
!!      FTS.DocID := IntArray[i];
!      FTS.Subject := aClient.OneFieldValue(TSQLRecordPeople,'FirstName',FTS.DocID);
!      FTS.Body := FTS.Subject+' bodY'+IntToStr(FTS.DocID);
!      aClient.Add(FTS,true);
!    end;
!    aClient.Commit; // Commit must be BEFORE OptimizeFTS3, memory leak otherwise
!    Check(FTS.OptimizeFTS3Index(Client.fServer));
The steps above are just typical. The only difference with a "standard" @*ORM@ approach is that the {\f1\fs20 DocID} property must be set {\i before} adding the {\f1\fs20 TSQLRecordFTS5} instance: there is no ID automatically created by {\i SQLite}, but an ID must be specified in order to link the FTS record to the original {\f1\fs20 TSQLRecordPeople} row, from its ID.
To support full-text queries, FTS maintains an inverted index that maps from each unique term or word that appears in the dataset to the locations in which it appears within the table contents. The dedicated {\f1\fs20 OptimizeFTS3Index} method is called to merge all existing index b-trees into a single large b-tree containing the entire index - this method will work with FTS3, FTS4 and FTS5 classes, whatever its name states. This can be an expensive operation, but may speed up future queries: you should not call this method after every modification of the FTS tables, but after some text has been added.
Then the FTS search query will use the custom {\f1\fs20 FTSMatch} method:
!  Check(aClient.FTSMatch(TSQLFTSTest,'Subject MATCH ''salVador1''',IntResult));
The matching IDs are stored in the {\f1\fs20 IntResult} integer {\i dynamic array}. Note that you can use a regular @*SQL@ query instead. Use of the {\f1\fs20 FTSMatch} method is not mandatory: in fact, it is just a wrapper around the {\f1\fs20 OneFieldValues} method, just using the "neutral" {\i RowID} column name for the results:
!function TSQLRest.FTSMatch(Table: TSQLRecordFTS3Class;
!  const WhereClause: RawUTF8; var DocID: TIntegerDynArray): boolean;
!begin // FTS3 tables do not have any ID, but RowID or DocID
!  result := OneFieldValues(Table,'RowID',WhereClause,DocID);
!end;
An overloaded {\f1\fs20 FTSMatch} method has been defined, and will handle detailed matching information, able to use a ranking algorithm. With this method, the results will be sorted by relevance:
!  Check(aClient.FTSMatch(TSQLFTSTest,'body1*',IntResult,[1,0.5]));
This method expects some additional constant parameters for weighting each FTS table column (there must be the same number of {\f1\fs20 PerFieldWeight} parameters as there are columns in the {\f1\fs20 TSQLRecordFTS5} table). In the above sample code, the {\f1\fs20 Subject} field will have a weight of 1.0, and he {\f1\fs20 Body} will be weighted as 0.5, i.e. any match in the '{\i body}' column content will be ranked twice less than any match in the '{\i subject}', which is probably of higher density.
The above query will call the following SQL statement:
$ SELECT RowID FROM FTSTest WHERE FTSTest MATCH 'body1*'
$ ORDER BY rank(matchinfo(FTSTest),1.0,0.5) DESC
The {\f1\fs20 rank} internal @*SQL function@ has been implemented in {\i Delphi}, following the guidelines of the official {\i SQLite3} documentation - as available from their Internet web site at @http://www.sqlite.org/fts3.html#appendix_a - to implement the most efficient way of implementing ranking. It will return the {\f1\fs20 RowID} of documents that match the full-text query sorted from most to least relevant. When calculating relevance, query term instances in the '{\i subject}' column are given twice the weighting of those in the '{\i body}' column.
:144   FTS4 index tables without content
Just as {\i SQlite3} allows, the framework permits FTS4 to forego storing the text being indexed, letting the indexed documents be stored in a database table created and managed by the user (an "external content" FTS4 table).
Because the indexed documents themselves are usually much larger than the full-text index, this option can be used to achieve significant storage space savings. Contentless FTS4 tables still support {\f1\fs20 SELECT} statements. However, it is an error to attempt to retrieve the value of any table column other than the {\f1\fs20 docid} column. The auxiliary function {\f1\fs20 matchinfo()} may be used - so {\f1\fs20 TSQLRest.FTSMatch} method will work as expected, but {\f1\fs20 snippet()} and {\f1\fs20 offsets()} will cause an exception at execution.
For instance, in sample "{\i 30 - MVC Server}", we define those two tables:
!  TSQLArticle = class(TSQLContent)
!  private
!    fContent: RawUTF8;
!    fTitle: RawUTF8;
!    fAbstract: RawUTF8;
!    fPublishedMonth: Integer;
!    fTags: TIntegerDynArray;
!  published
!    property Title: RawUTF8 index 80 read fTitle write fTitle;
!    property Content: RawUTF8 read fContent write fContent;
!    property PublishedMonth: Integer read fPublishedMonth write fPublishedMonth;
!    property Abstract: RawUTF8 index 1024 read fAbstract write fAbstract;
!    property Tags: TIntegerDynArray index 1 read fTags write fTags;
!  end;
!
!  TSQLArticleSearch = class(TSQLRecordFTS4Porter)
!  private
!    fContent: RawUTF8;
!    fTitle: RawUTF8;
!    fAbstract: RawUTF8;
!  published
!    property Title: RawUTF8 read fTitle write fTitle;
!    property Abstract: RawUTF8 read fAbstract write fAbstract;
!    property Content: RawUTF8 read fContent write fContent;
!  end;
And we initialized the database model to let all data be stored only in {\f1\fs20 TSQLArticle}, not in {\f1\fs20 TSQLArticleSearch}, using an "external content" FTS4 table to index the text from the selected {\f1\fs20 Title}, {\f1\fs20 Abstract} and {\f1\fs20 Content} fields of {\f1\fs20 TSQLArticle}:
!function CreateModel: TSQLModel;
!begin
!  result := TSQLModel.Create([TSQLBlogInfo,TSQLAuthor,
!    TSQLTag,TSQLArticle,TSQLComment,TSQLArticleSearch],'blog');
!!  result.Props[TSQLArticleSearch].FTS4WithoutContent(TSQLArticle);
!  ...
The {\f1\fs20 TSQLModelRecordProperties.FTS4WithoutContent()} will in fact create the needed {\i SQLite3} triggers, to automatically populate the {\f1\fs20 ArticleSearch} Full Text indexes when the main {\f1\fs20 Article} row changes.
Since this FTS4 feature is specific to {\i SQlite3}, and triggers do not work on virtual tables (by now), this method won't do anything if the {\f1\fs20 TSQLArticleSearch} or {\f1\fs20 TSQLArticle} are on an external database - see @27@. Both need to be stored in the main {\i SQLite3} DB.
In the {\i 30 - MVC Server} sample, the search will be performed as such:
!  if scop^.GetAsRawUTF8('match',match) and fHasFTS then begin
!    if scop^.GetAsDouble('lastrank',rank) then
!      whereClause := 'and rank<? ';
!!    whereClause := 'join (select docid,rank(matchinfo(ArticleSearch),1.0,0.7,0.5) as rank '+
!!      'from ArticleSearch where ArticleSearch match ? '+whereClause+
!!      'order by rank desc limit 100) as r on (r.docid=Article.id)';
!    articles := RestModel.RetrieveDocVariantArray(
!      TSQLArticle,'',whereClause,[match,rank],
!      'id,title,tags,author,authorname,createdat,abstract,contenthtml,rank');
In the above query expression, the {\f1\fs20 rank()} function is used over the detailed FTS4 search statistics returned by {\f1\fs20 matchinfo()}, using a 1.0 weight for any match in the {\f1\fs20 Title} column, 0.7 for the {\f1\fs20 Abstract} column, and 0.5 for {\f1\fs20 Content}. The matching articles content is then returned in an {\f1\fs20 articles:} {\f1\fs20 TDocVariant} array, ready to be rendered on the web page.
:  Column collations
In any database, there is a need to define how column data is to be compared. It is needed for proper search and ordering of the data. This is the purpose of so-called {\i @**collation@s}.
By default, when {\i SQLite} compares two strings, it uses a collating sequence or collating function (two words for the same thing) to determine which string is greater or if the two strings are equal. {\i SQLite} has three built-in collating functions: BINARY, NOCASE, and RTRIM:
- BINARY - Compares string data using {\f1\fs20 memcmp()}, regardless of text encoding.
- NOCASE - The same as binary, except the 26 upper case characters of ASCII are folded to their lower case equivalents before the comparison is performed. Note that only ASCII characters are case folded. Plain {\i SQLite} does not attempt to do full @*Unicode@ case folding due to the size of the tables required - but you could use {\i mORMot}'s SYSTEMNOCASE, or WIN32CASE/WIN32NOCASE custom collations for enhanced case folding support (see below);
- RTRIM - The same as binary, except that trailing space characters are ignored.
In the {\i mORMot} ORM, we defined some additional kind of collations, via some internal calls to the {\f1\fs20 sqlite3_create_collation()} API:
|%25%60
|\b TSQLFieldType|Default collation\b0
|{\f1\fs20 sftAnsiText}|NOCASE
|{\f1\fs20 sftUTF8Text}|SYSTEMNOCASE, i.e. using {\f1\fs20 UTF8ILComp()}, which will ignore {\i Win-1252} Latin accents
|{\f1\fs20 sftEnumerate\line sftSet\line sftInteger\line sftID\line sftTID\line sftRecord\line sftBoolean\line sftFloat\line sftCurrency\line ftTimeLog\line sftModTime\line sftCreateTime}|BINARY is used for those numerical values
|{\f1\fs20 sftDateTime\line ftDateTimeMS}|ISO8601, i.e. decoding the text into a date/time value before comparison
|{\f1\fs20 sftObject\line sftVariant}|BINARY, since it is stored as plain JSON content
|{\f1\fs20 sftBlob\line sftBlobDynArray\line sftBlobCustom}|BINARY
;|{\f1\fs20 sftUTF8Custom}|NOCASE by default
|%
You can override those default collation schemes by calling {\f1\fs20 TSQLRecordProperties. SetCustomCollationForAll()} (which will override all fields collation for a given type) or {\f1\fs20 SetCustomCollation()} method (which will override a given field) in an overridden {\f1\fs20 class procedure InternalRegisterCustomProperties()} or {\f1\fs20 InternalDefineModel()}, so that it will be common to all database models, for both client and server, every time the corresponding {\f1\fs20 TSQLRecord} is used.
As an alternative, you may call {\f1\fs20 TSQLModel.SetCustomCollationForAll()} method, which will do it for a given model.
The following collations are therefore available when using {\i SQLite3} within the {\i mORMot} ORM:
|%25%70
|\b Collation|Description\b0
|BINARY|Default {\f1\fs20 memcmp()} comparison
|NOCASE|Default ASCII 7 bit comparison
|RTRIM|Default {\f1\fs20 memcmp()} comparison with right trim
|SYSTEMNOCASE|{\i mORMot}'s Win-1252 8 bit comparison
;|UNICODENOCASE|{\i mORMot}'s Unicode 10.0 comparison
|ISO8601|{\i mORMot}'s date/time comparison
|WIN32CASE|{\i mORMot}'s comparison using case-insensitive Windows API
|WIN32NOCASE|{\i mORMot}'s comparison using not case-insensitive Windows API
|%
Note that WIN32CASE/WIN32NOCASE will be slower than the others, but will handle properly any kind of complex scripting. For instance, if you want to use the Unicode-ready Windows API at database level, you can set for each database model:
! aModel.SetCustomCollationForAll(sftUTF8Text,'WIN32CASE');
! aModel.SetCustomCollationForAll(sftDateTime,'NOCASE');
On non-Windows platform, it will either use the system ICU library (if available), or fallback to the FPC RTL with temporary {\f1\fs20 UnicodeString} values - which requires to include `cwstrings` in your project uses clause. Note that depending on the library used, the results may not be consistent: so if you move a {\i SQLite3} database file e.g. from a Windows system to a Linux system with WIN32CASE collation, you should better regenerate all your indexes!
If you use non-default collations (i.e. SYSTEMNOCASE/ISO8601/WIN32CASE/WIN32NOCASE), you may have trouble running requests with "plain" {\i SQLite3} tools. But you can use our {\f1\fs20 @*SynDBExplorer@} safely, since it will declare all the above collations.
When using external databases - see @27@, if the content is retrieved directly from the database driver and by-passes the virtual table mechanism - see @20@, returned data may not match your expectations according to the custom collations: you will need to customize the external tables definition by hand, with the proper SQL statement of each external DB engine.
Note that {\i @*mORMot 2@} offers a new UNICODENOCASE collation, which follows Unicode 10.0 without any Windows or ICU API call, so is consistent on all systems - and is also faster.
:  REGEXP operator
Our {\i SQLite3} engine can use {\i @**regular expression@} within its SQL queries, by enabling the {\f1\fs20 @**REGEXP@} operator in addition to standard SQL operators ({\f1\fs20 =  == != <> IS IN LIKE GLOB MATCH}).
:   Default REGEXP Engine
By default, and since mORMot 1.18.6218 (25 January 2021), our static {\i SQlite3} engine includes a compact and efficient enough C extension, as available from the official {\i SQLite3} project source code tree. It is included with the official amalgamation file during our compilation phase.
So you don't need to do anything to be able to use the REGEX operator in your queries:
!Server := TSQLRestServerDB.Create(Model,'test.db3');
!try
!  with TSQLRecordPeople.CreateAndFillPrepare(Client,
!!    'FirstName REGEXP ?',['\bFinley\b']) do
!  try
!    while FillOne do begin
!      Check(LastName='Morse');
!      Check(IdemPChar(pointer(FirstName),'SAMUEL FINLEY '));
!    end;
!  finally
!    Free;
!  end;
!finally
!  Server.Free;
!end;
The above code will execute the following SQL statement (with a prepared parameter for the regular expression itself):
! SELECT * from People WHERE Firstname REGEXP '\bFinley\b';
That is, it will find all objects where {\f1\fs20 TSQLRecordPeople.FirstName} will contain the {\f1\fs20 'Finley'} word - in a regular expression, {\f1\fs20 \\b} defines a word {\f1\fs20 b}oundary search.
In fact, the {\f1\fs20 REGEXP} operator is a special syntax for the {\f1\fs20 regexp()} user function. No {\f1\fs20 regexp()} user function is defined by default and so use of the {\f1\fs20 REGEXP} operator will normally result in an error message. Calling {\f1\fs20 CreateRegExFunction()} for a given connection will add a SQL function named "{\f1\fs20 regexp()}" at run-time, which will be called in order to implement the {\f1\fs20 REGEXP} operator.
:   PCRE REGEXP Engine
If you want to use the Open Source PCRE library to perform the searches, instead of this default C extension, you should include the {\f1\fs20 SynSQLite3RegEx.pas} unit to your uses clause, and register the {\f1\fs20 RegExp()} SQL function to a given {\i SQLite3} database instance, as such:
!uses SynCommons, mORmot, mORMotSQLite3,
!!  SynSQLite3RegEx;
! ...
!Server := TSQLRestServerDB.Create(Model,'test.db3');
!try
!!  CreateRegExpFunction(Server.DB.DB);
!  with TSQLRecordPeople.CreateAndFillPrepare(Client,
!!    'FirstName REGEXP ?',['\bFinley\b']) do
!  try
!    while FillOne do begin
!      Check(LastName='Morse');
! ...
It will use the statically linked PCRE library as available since {\i Delphi} XE, or will rely on the {\f1\fs20 PCRE.pas} wrapper unit as published at @http://www.regular-expressions.info/download/TPerlRegEx.zip for older versions of {\i Delphi}.
This unit will call directly the @*UTF-8@ API of the PCRE library, and maintain a per-connection cache of compiled regular expressions to ensure the best performance possible.
:60  ACID and speed
As stated above in @59@, the default {\i SQlite3} write speed is quite slow, when running on a normal hard drive. By default, the engine will pause after issuing a OS-level write command. This guarantees that the data is written to the disk, and features the @*ACID@ properties of the database engine.
{\f1\fs20 @**ACID@} is an acronym for "{\i Atomicity Consistency Isolation Durability}" properties, which guarantee that database transactions are processed reliably: for instance, in case of a power loss or hardware failure, the data will be saved on disk in a consistent way, with no potential loss of data.
In {\i SQLite3}, ACID is implemented by two means at file level:
- {\i Synchronous writing}: it means that the engine will wait for any written content to be flushed to disk before processing the next request;
- {\i File locking}: it means that the database file is locked for exclusive use during writing, allowing several processes to access the same database file concurrently.
Changing these default settings can ensure much better writing performance.
:   Synchronous writing
You can overwrite the default ACID behavior by setting the {\f1\fs20 TSQLDataBase.Synchronous} property to {\f1\fs20 smOff} instead of the default {\f1\fs20 smFull} setting. When {\f1\fs20 Synchronous} is set to {\f1\fs20 smOff}, {\i SQLite} continues without syncing as soon as it has handed data off to the operating system. If the application running {\i SQLite} crashes, the data will be safe, but the database might become corrupted if the operating system crashes or the computer loses power before that data has been written to the disk surface. On the other hand, some operations are as much as 50 or more times faster with this setting.
When the tests performed during @59@ use {\f1\fs20 Synchronous := smOff}, "Write one" speed is enhanced from 8-9 rows per second into about 400 rows per second, on a physical hard drive (SSD or NAS drives may not suffer from this delay).
So depending on your application requirements, you may switch Synchronous setting to off.
To change the main {\i SQLite3} engine synchronous parameter, you may code for instance:
!Client := TSQLRestClientDB.Create(Model,nil,MainDBFileName,TSQLRestServerDB,false,'');
!!Client.Server.DB.Synchronous := smOff;
Note that this setting is common to a whole {\f1\fs20 TSQLDatabase} instance, so will affect all tables handled by the {\f1\fs20 @*TSQLRestServerDB@} instance.
But if you defined some {\i SQLite3} external tables - see @27@, you can define the setting for a particular external connection, for instance:
!Props := TSQLDBSQLite3ConnectionProperties.Create(DBFileName,'''','');
!VirtualTableExternalRegister(Model,TSQLRecordSample,Props,'SampleRecord');
!Client := TSQLRestClientDB.Create(Model,nil,MainDBFileName,TSQLRestServerDB,false,'');
!!TSQLDBSQLite3Connection(Props.MainConnection).Synchronous := smOff;
Never forget that you may have several {\i SQlite3} engines within a single {\i mORMot} server!
:   File locking
You can overwrite the first default ACID behavior by setting the {\f1\fs20 TSQLDataBase.LockingMode} property to {\f1\fs20 lmExclusive} instead of the default {\f1\fs20 lmNormal} setting. When {\f1\fs20 LockingMode} is set to {\f1\fs20 lmExclusive}, {\i SQLite} will lock the database file for exclusive use during the whole session. It will prevent other processes (e.g. database viewer tools) to access the file at the same time, but small write transactions will be much faster, by a factor usually greater than 40. Bigger transactions involving several hundredths/thousands of INSERT won't be accelerated - but individual insertions will have a major speed up - see @59@.
To change the main {\i SQLite3} engine locking mode parameter, you may code for instance:
!Client := TSQLRestClientDB.Create(Model,nil,MainDBFileName,TSQLRestServerDB,false,'');
!!Client.Server.DB.LockingMode := lmExclusive;
Note that this setting is common to a whole {\f1\fs20 TSQLDatabase} instance, so will affect all tables handled by the {\f1\fs20 @*TSQLRestServerDB@} instance.
But if you defined some {\i SQLite3} external tables - see @27@, you can define the setting for a particular external connection, for instance:
!Props := TSQLDBSQLite3ConnectionProperties.Create(DBFileName,'''','');
!VirtualTableExternalRegister(Model,TSQLRecordSample,Props,'SampleRecord');
!Client := TSQLRestClientDB.Create(Model,nil,MainDBFileName,TSQLRestServerDB,false,'');
!!TSQLDBSQLite3Connection(Props.MainConnection).LockingMode := lmExclusive;
In fact, exclusive file locking improves the reading speed by a factor of 4 (in case of individual row retrieval). As such, defining {\f1\fs20 LockingMode := lmExclusive} without {\f1\fs20 Synchronous := smOff} could be of great benefit for a server which purpose is mainly to serve ORM content to clients.
:   Performance tuning
By default, the slow but truly {\f1\fs20 ACID} setting will be used with {\i mORMot}, just as with {\i SQlite3}. We do not change this policy, since it will ensure best safety, in the expense of slow writing outside a transaction.
The best performance will be achieved by combining the two previous options, as such:
!Client := TSQLRestClientDB.Create(Model,nil,MainDBFileName,TSQLRestServerDB,false,'');
!!Client.Server.DB.LockingMode := lmExclusive;
!!Client.Server.DB.Synchronous := smOff;
Or, for external tables:
!Props := TSQLDBSQLite3ConnectionProperties.Create(DBFileName,'''','');
!VirtualTableExternalRegister(Model,TSQLRecordSample,Props,'SampleRecord');
!Client := TSQLRestClientDB.Create(Model,nil,MainDBFileName,TSQLRestServerDB,false,'');
!!TSQLDBSQLite3Connection(Props.MainConnection).Synchronous := smOff;
!!TSQLDBSQLite3Connection(Props.MainConnection).LockingMode := lmExclusive;
If you can afford loosing some data in very rare border case, or if you are sure your hardware configuration is safe (e.g. if the server is connected to a power inverter and has RAID disks) and that you have backups at hand, setting {\f1\fs20 Synchronous := smOff} will help your application scale for writing. Setting {\f1\fs20 LockingMode := lmExclusive} will benefit of both writing and reading speed. Consider using an external and dedicated database (like {\i @*Firebird@}, {\i @*Oracle@, @*PostgreSQL@, @*MySQL@, @*DB2@, @*Informix@} or @*MS SQL@) if your security expectations are very high, and if the default safe but slow setting is not enough for you.
:  Database backup
In all cases, do not forget to perform @*backup@s of your {\i SQlite3} database as often as possible (at least several times a day). Adding a backup feature on the server side is as simple as running:
! Server.DB.BackupBackground('backup.db3',1024,10,nil);
The above line will perform a background live backup of the main {\i SQLite3} database, by steps of 1024 pages (i.e. it will process 1 MB per step, since default page size is 1024 bytes), performing a little sleep of 10 milliseconds between each 1 MB copy step, allowing main CRUD / ORM operations to continue uninterrupted during the backup.\line You can even specify an {\f1\fs20 OnProgress: TSQLDatabaseBackupEvent} callback event, to monitor the backup process.
Note that {\f1\fs20 @*TSQLRestServerDB@.Backup} or {\f1\fs20 TSQLRestServerDB.BackupGZ} methods are not recommended any more on a running {\i mORMot} database, due to some potential issues with virtual tables, especially on the {\i Win64} platform. You should definitively use {\f1\fs20 TSQLDatabase.BackupBackground()} instead.
The same backup process can be used e.g. to save an in-memory {\i SQLite3} database into a {\i SQLite3} file, as such:
! if aInMemoryDB.BackupBackground('backup.db3',-1,0,nil) then
!   aInMemoryDB.BackupBackgroundWaitUntilFinished;
Above code will save the {\f1\fs20 aInMemoryDB} database into the '{\f1\fs20 backup.db3}' file.
\page
:20 Virtual Tables magic
The {\i @*SQlite3@} engine has the unique ability to create @**Virtual Table@s from code. From the perspective of an @*SQL@ statement, the virtual table object looks like any other table or view. But behind the scenes, queries from and updates to a virtual table invoke callback methods on the virtual table object instead of reading and writing to the database file.
The virtual table mechanism allows an application to publish interfaces that are accessible from SQL statements as if they were tables. SQL statements can in general do anything to a virtual table that they can do to a real table, with the following exceptions:
- One cannot create a trigger on a virtual table.
- One cannot create additional indices on a virtual table. (Virtual tables can have indices but that must be built into the virtual table implementation. Indices cannot be added separately using {\f1\fs20 CREATE INDEX} statements.)
- One cannot run {\f1\fs20 ALTER TABLE ... ADD COLUMN} commands against a virtual table.
- Particular virtual table implementations might impose additional constraints. For example, some virtual implementations might provide read-only tables. Or some virtual table implementations might allow {\f1\fs20 INSERT} or {\f1\fs20 DELETE} but not {\f1\fs20 UPDATE}. Or some virtual table implementations might limit the kinds of {\f1\fs20 UPDATE}s that can be made.
Example of virtual tables, already included in the {\i SQLite3} engine, are @*FTS@ or @*RTREE@ tables.
Our framework introduces new types of custom virtual table. You'll find classes like {\f1\fs20 @*TSQLVirtualTableJSON@} or {\f1\fs20 @*TSQLVirtualTableBinary@} which handle in-memory data structures. Or it might represent a view of data on disk that is not in the {\i SQLite3} format (e.g. {\f1\fs20 TSQLVirtualTableLog}). It can be used to access any external database, just as if they were native {\i SQLite3} tables - see @27@. Or the application might compute the content of the virtual table on demand.
Thanks to the generic implementation of Virtual Table in {\i SQLite3}, you can use such tables in your SQL statement, and even safely execute a {\f1\fs20 SELECT} statement with {\f1\fs20 @*JOIN@} or custom functions, mixing normal {\i SQLite3} tables and any other Virtual Table. From the @*ORM@ point of view, virtual tables are just tables, i.e. they inherit from {\f1\fs20 TSQLRecordVirtual}, which inherits from the common base {\f1\fs20 TSQLRecord} class.
:  Virtual Table module classes
A dedicated mechanism has been added to the framework, beginning with revision 1.13, in order to easily add such virtual tables with pure {\i Delphi} code.
In order to implement a new @*Virtual Table@ type, you'll have to define a so called {\i Module} to handle the fields and data access and an associated {\i Cursor} for the {\f1\fs20 SELECT} statements. This is implemented by the two {\f1\fs20 TSQLVirtualTable} and {\f1\fs20 TSQLVirtualTableCursor} classes as defined in the @!TSQLVirtualTable,TSQLVirtualTableCursor,TSQLVirtualTableJSON,TSQLVirtualTableBinary,TSQLVirtualTableLog,TSQLVirtualTableCursorLog,TSQLVirtualTableCursorJSON,TSQLVirtualTableCursorIndex!Lib\SQLite3\mORMot.pas@ unit.
For instance, here are the default Virtual Table classes deriving from those classes:
\graph HierTSQLVirtualTable Virtual Tables classes hierarchy
\TSQLVirtualTableJSON\TSQLVirtualTable
\TSQLVirtualTableBinary\TSQLVirtualTableJSON
\TSQLVirtualTableLog\TSQLVirtualTable
\TSQLVirtualTableCursorIndex\TSQLVirtualTableCursor
\TSQLVirtualTableCursorJSON\TSQLVirtualTableCursorIndex
\TSQLVirtualTableCursorLog\TSQLVirtualTableCursorIndex
\
{\f1\fs20 @*TSQLVirtualTableJSON@, @*TSQLVirtualTableBinary@} and {\f1\fs20 TSQLVirtualTableCursorJSON} classes will implement a Virtual Table using a {\f1\fs20 @*TSQLRestStorageInMemory@} instance to handle fast in-memory @*static@ databases. Disk storage will be encoded either as @*UTF-8@ @*JSON@ (for the {\f1\fs20 TSQLVirtualTableJSON} class, i.e. the '{\f1\fs20 JSON}' module), or in a proprietary @*SynLZ@ compressed format (for the {\f1\fs20 TSQLVirtualTableBinary} class, i.e. the '{\f1\fs20 Binary}' module). File extension on disk will be simply {\f1\fs20 .json} for the '{\f1\fs20 JSON}' module, and {\f1\fs20 .data} for the '{\f1\fs20 Binary}' module. Just to mention the size on disk difference, the 502 KB {\f1\fs20 People.json} content (as created by included regression tests) is stored into a 92 KB {\f1\fs20 People.data} file, in our proprietary optimized format.
Note that the virtual table module name is retrieved from the class name. For instance, the {\f1\fs20 TSQLVirtualTableJSON} class will have its module named as 'JSON' in the SQL code.
To handle external databases, two dedicated classes, named {\f1\fs20 TSQLVirtualTableExternal} and {\f1\fs20 TSQLVirtualTableCursorExternal} will be defined in a similar manner - see @%%HierExternalTables@ @30@.
As you probably have already stated, all those Virtual Table mechanism is implemented in {\f1\fs20 mORMot.pas}. Therefore, it is independent from the {\i @*SQLite3@} engine, even if, to my knowledge, there is no other SQL database engine around able to implement this pretty nice feature.
:  Defining a Virtual Table module
Here is how the {\f1\fs20 TSQLVirtualTableLog} class type is defined, which will implement a @*Virtual Table@ module named "{\f1\fs20 Log}". Note that the {\i SQLite3} virtual table module name will be computed from the class name, trimming its first characters, e.g. {\f1\fs20 TSQLVirtualTable{\b Log}} will trim trailing {\f1\fs20 TSQLVirtualTable} and define a {\f1\fs20 'Log'} virtual module.
Adding a new module is just made by overriding some {\i Delphi} methods:
!  TSQLVirtualTableLog = class(TSQLVirtualTable)
!  protected
!    fLogFile: TSynLogFile;
!  public
!    class procedure GetTableModuleProperties(
!      var aProperties: TVirtualTableModuleProperties); override;
!    constructor Create(aModule: TSQLVirtualTableModule; const aTableName: RawUTF8;
!      FieldCount: integer; Fields: PPUTF8CharArray); override;
!    destructor Destroy; override;
!  end;
This module will allow direct Read-Only access to a {\f1\fs20 .log} file content, which file name will be specified by the corresponding SQL table name.
The following method will define the properties of this Virtual Table Module:
!class procedure TSQLVirtualTableLog.GetTableModuleProperties(
!  var aProperties: TVirtualTableModuleProperties);
!begin
!  aProperties.Features := [vtWhereIDPrepared];
!  aProperties.CursorClass := TSQLVirtualTableCursorLog;
!  aProperties.RecordClass := TSQLRecordLogFile;
!end;
The supplied feature set defines a read-only module (since {\f1\fs20 vtWrite} is not selected), and {\f1\fs20 vtWhereIDPrepared} indicates that any {\f1\fs20 RowID=?} SQL statement will be handled as such in the cursor class (we will use the log row as ID number, start counting at 1, so we can speed up {\f1\fs20 RowID=?} WHERE clause easily). The associated cursor class is returned. And a {\f1\fs20 @*TSQLRecord@} class is specified, to define the handled fields - its @*published properties@ definition will be used by the inherited {\f1\fs20 Structure} method to specify to the {\i @*SQLite3@} engine which kind of fields are expected in the SQL statements:
!  TSQLRecordLogFile = class(TSQLRecordVirtualTableAutoID)
!  protected
!    fContent: RawUTF8;
!    fDateTime: TDateTime;
!    fLevel: TSynLogInfo;
!  published
!    /// the log event time stamp
!    property DateTime: TDateTime read fDateTime;
!    /// the log event level
!    property Level: TSynLogInfo read fLevel;
!    /// the textual message associated to the log event
!    property Content: RawUTF8 read fContent;
!  end;
You could have overridden the {\f1\fs20 Structure} method in order to provide the {\f1\fs20 CREATE TABLE} SQL statement expected. But using {\i Delphi} class RTTI allows the construction of this SQL statement with the appropriate column type and @*collation@, common to what the rest of the @*ORM@ will expect.
Of course, this {\f1\fs20 RecordClass} property is not mandatory. For instance, the {\f1\fs20 TSQLVirtualTableJSON.GetTableModuleProperties} method won't return any associated {\f1\fs20 TSQLRecordClass}, since it will depend on the table it is implementing, i.e. the running {\f1\fs20 @*TSQLRestStorageInMemory@} instance. Instead, the {\f1\fs20 Structure} method is overridden, and will return the corresponding field layout of each associated table.
Here is how the {\f1\fs20 Prepare} method is implemented, and will handle the {\f1\fs20 vtWhereIDPrepared} feature:
!function TSQLVirtualTable.Prepare(var Prepared: TSQLVirtualTablePrepared): boolean;
!begin
!  result := Self<>nil;
!  if result then
!!    if (vtWhereIDPrepared in fModule.Features) and
!!       Prepared.IsWhereIDEquals(true) then
!    with Prepared.Where[0] do begin // check ID=?
!      Value.VType := varAny; // mark TSQLVirtualTableCursorJSON expects it
!      OmitCheck := true;
!      Prepared.EstimatedCost := 1;
!    end else
!      Prepared.EstimatedCost := 1E10; // generic high cost
!end;
Then here is how each '{\f1\fs20 log}' virtual table module instance is created:
!constructor TSQLVirtualTableLog.Create(aModule: TSQLVirtualTableModule;
!  const aTableName: RawUTF8; FieldCount: integer; Fields: PPUTF8CharArray);
!var aFileName: TFileName;
!begin
!  inherited;
!  if (FieldCount=1) then
!    aFileName := UTF8ToString(Fields[0]) else
!    aFileName := aModule.FileName(aTableName);
!  fLogFile := TSynLogFile.Create(aFileName);
!end;
It only associates a {\f1\fs20 TSynLogFile} instance according to the supplied file name (our SQL {\f1\fs20 CREATE VIRTUAL TABLE} statement only expects one parameter, which is the {\f1\fs20 .log} file name on disk - if this file name is not specified, it will use the SQL table name instead).
The {\f1\fs20 TSQLVirtualTableLog.Destroy} destructor will free this {\f1\fs20 fLogFile} instance:
!destructor TSQLVirtualTableLog.Destroy;
!begin
!  FreeAndNil(fLogFile);
!  inherited;
!end;
Then the corresponding cursor is defined as such:
!  TSQLVirtualTableCursorLog = class(TSQLVirtualTableCursorIndex)
!  public
!    function Search(const Prepared: TSQLVirtualTablePrepared): boolean; override;
!    function Column(aColumn: integer; var aResult: TVarData): boolean; override;
!  end;
Since this class inherits from {\f1\fs20 TSQLVirtualTableCursorIndex}, it will have the generic {\f1\fs20 fCurrent / fMax} protected fields, and will have the {\f1\fs20 HasData, Next} and {\f1\fs20 Search} methods using those properties to handle navigation throughout the cursor.
The overridden {\f1\fs20 Search} method consists only in:
!function TSQLVirtualTableCursorLog.Search(
!  const Prepared: TSQLVirtualTablePrepared): boolean;
!begin
!  result := inherited Search(Prepared); // mark EOF by default
!  if result then begin
!    fMax := TSQLVirtualTableLog(Table).fLogFile.Count-1;
!    if Prepared.IsWhereIDEquals(false) then begin
!      fCurrent := Prepared.Where[0].Value.VInt64-1; // ID=? -> index := ID-1
!      if cardinal(fCurrent)<=cardinal(fMax) then
!        fMax := fCurrent else // found one
!        fMax := fCurrent-1;   // out of range ID
!    end;
!  end;
!end;
The only purpose of this method is to handle {\f1\fs20 RowID=?} statement {\f1\fs20 SELECT WHERE} clause, returning {\f1\fs20 fCurrent=fMax=ID-1} for any valid {\f1\fs20 ID}, or {\f1\fs20 fMax<fCurrent}, i.e. no result if the {\f1\fs20 ID} is out of range. In fact, the {\f1\fs20 Search} method of the cursor class must handle all cases which has been notified as handled during the call to the {\f1\fs20 Prepare} method. In our case, since we have set the {\f1\fs20 vtWhereIDPrepared} feature and the {\f1\fs20 Prepare} method identified it in the request and set the {\f1\fs20 OmitCheck} flag, our {\f1\fs20 Search} method MUST handle the {\f1\fs20 RowID=?} case.
If the {\f1\fs20 WHERE} clause is not {\f1\fs20 RowID=?} (i.e. if {\f1\fs20 Prepared.IsWhereIDEquals} returns false), it will return {\f1\fs20 fCurrent=0} and {\f1\fs20 fMax=fLogFile.Count-1}, i.e. it will let the {\i SQLite3} engine loop through all rows searching for the data.
Each column value is retrieved by this method:
!function TSQLVirtualTableCursorLog.Column(aColumn: integer;
!  var aResult: TVarData): boolean;
!var LogFile: TSynLogFile;
!begin
!  result := false;
!  if (self=nil) or (fCurrent>fMax) then
!    exit;
!  LogFile := TSQLVirtualTableLog(Table).fLogFile;
!  if LogFile=nil then
!    exit;
!  case aColumn of
!   -1: SetColumn(aResult,fCurrent+1); // ID = index + 1
!    0: SetColumn(aResult,LogFile.EventDateTime(fCurrent));
!    1: SetColumn(aResult,ord(LogFile.EventLevel[fCurrent]));
!    2: SetColumn(aResult,LogFile.LinePointers[fCurrent],LogFile.LineSize(fCurrent));
!    else exit;
!  end;
!  result := true;
!end;
As stated by the documentation of the {\f1\fs20 TSQLVirtualTableCursor} class, {\f1\fs20 -1} is the column index for the {\f1\fs20 RowID}, and then will follow the columns as defined in the text returned by the {\f1\fs20 Structure} method (in our case, the {\f1\fs20 DateTime, Level, Content} fields of {\f1\fs20 TSQLRecordLogFile}).
The {\f1\fs20 SetColumn} overloaded methods can be used to set the appropriate result to the {\f1\fs20 aResult} variable. For @*UTF-8@ text, it will use a temporary in-memory space, to ensure that the text memory will be still available at least until the next {\f1\fs20 Column} method call.
:  Using a Virtual Table module
From the low-level {\i @*SQLite3@} point of view, here is how this "{\f1\fs20 Log}" @virtual table@ module can be used, directly from the {\i SQLite3} engine.
First we will register this module to a DB connection (this method is to be used only in case of such low-level access - in our @*ORM@ you should never call this method, but {\f1\fs20 TSQLModel. VirtualTableRegister} instead, {\i cf.} next paragraph):
! RegisterVirtualTableModule(TSQLVirtualTableLog,Demo);
Then we can execute the following SQL statement to create the virtual table for the {\f1\fs20 Demo} database connection:
! Demo.Execute('CREATE VIRTUAL TABLE test USING log(temptest.log);');
This will create the virtual table. Since all fields are already known by the {\f1\fs20 TSQLVirtualTableLog} class, it is not necessary to specify the fields at this level. We only specify the log file name, which will be retrieved by {\f1\fs20 TSQLVirtualTableLog. Create} constructor.
! Demo.Execute('select count(*) from test',Res);
! Check(Res=1);
! s := Demo.ExecuteJSON('select * from test');
! s2 := Demo.ExecuteJSON('select * from test where rowid=1');
! s3 := Demo.ExecuteJSON('select * from test where level=3');
You can note that there is no difference with a normal {\i SQLite3} table, from the SQL point of view. In fact, the full power of the SQL language as implemented by {\i SQLite3} - see @http://sqlite.org/lang.html - can be used with any kind of data, if you define the appropriate methods of a corresponding Virtual Table module.
:  Virtual Table, ORM and TSQLRecord
The framework @*ORM@ is able to use @*Virtual Table@ modules, just by defining some {\f1\fs20 @*TSQLRecord@}, inheriting from some {\f1\fs20 TSQLRecordVirtual} dedicated classes:
\graph HierTSQLRecordVirtualTableForcedID Custom Virtual Tables records classes hierarchy
\TSQLRecordVirtualTableAutoID\TSQLRecordVirtual
\TSQLRecordVirtual\TSQLRecord
\TSQLRecordLogFile\TSQLRecordVirtualTableAutoID
\TSQLRecordVirtualTableForcedID\TSQLRecordVirtual
\
{\f1\fs20 @*TSQLRecordVirtualTableAutoID@} children can be defined for Virtual Table implemented in {\i Delphi}, with a new {\f1\fs20 ID} generated automatically at {\f1\fs20 INSERT}.
{\f1\fs20 @*TSQLRecordVirtualTableForcedID@} children can be defined for Virtual Table implemented in {\i Delphi}, with an {\f1\fs20 ID} value forced at {\f1\fs20 INSERT} (in a similar manner than for {\f1\fs20 TSQLRecordRTree} or {\f1\fs20 TSQLRecordFTS3}/{\f1\fs20 FTS4}/{\f1\fs20 FTS5}).
{\f1\fs20 TSQLRecordLogFile} was defined to map the column name as retrieved by the {\f1\fs20 TSQLVirtualTableLog} ('{\f1\fs20 log}') module, and should not to be used for any other purpose.
The Virtual Table module associated from such classes is retrieved from an association made to the server {\f1\fs20 @*TSQLModel@}. In a @*Client-Server@ application, the association is not needed (nor to be used, since it may increase code size) on the Client side. But on the server side, the {\f1\fs20 TSQLModel. VirtualTableRegister} method must be called to associate a {\f1\fs20 TSQLVirtualTableClass} (i.e. a Virtual Table module implementation) to a {\f1\fs20 TSQLRecordVirtualClass} (i.e. its ORM representation).
For instance, the following code will register two {\f1\fs20 TSQLRecord} classes, the first using the '{\f1\fs20 JSON}' virtual table module, the second using the '{\f1\fs20 Binary}' module:
!  Model.VirtualTableRegister(TSQLRecordDali1,TSQLVirtualTableJSON);
!  Model.VirtualTableRegister(TSQLRecordDali2,TSQLVirtualTableBinary);
This registration should be done on the Server side only, {\i before} calling {\f1\fs20 TSQLRestServer.Create} (or {\f1\fs20 TSQLRestClientDB.Create}, for a @*stand-alone@ application). Otherwise, an exception is raised at virtual table creation.
:57  In-Memory "static" process
We have seen that the {\f1\fs20 @*TSQLVirtualTableJSON@, @*TSQLVirtualTableBinary@} and {\f1\fs20 TSQLVirtualTableCursorJSON} classes implement a @*Virtual Table@ module using a {\f1\fs20 @**TSQLRestStorageInMemory@} instance to handle fast @**static@ in-memory database.
Why use such a database type, when you can create a {\i @*SQLite3@} in-memory table, using the {\f1\fs20 :memory:} file name? That is the question...
- {\i SQlite3} in-memory tables are not persistent, whereas our {\f1\fs20 JSON} or {\f1\fs20 Binary} virtual table modules can be written on disk on purpose, if the {\f1\fs20 aServer.StaticVirtualTable[aClass].CommitShouldNotUpdateFile} property is set to {\f1\fs20 true} - in this case, file writing should be made by calling explicitly the {\f1\fs20 aServer.StaticVirtualTable[aClass].UpdateToFile} method;
- {\i SQlite3} in-memory tables will need two database connections, or call to the {\f1\fs20 @*ATTACH DATABASE@} SQL statement - both of them are not handled natively by our @*Client-Server@ framework;
- {\i SQlite3} in-memory tables are only accessed via SQL statements, whereas {\f1\fs20 @*TSQLRestStorageInMemory@} tables can have faster direct access for most common @*REST@ful commands ({\f1\fs20 GET / POST / PUT / DELETE} individual rows) - this could make a difference in server CPU load, especially with the @*Batch@ feature of the framework;
- On the server side, it could be very convenient to have a direct list of in-memory {\f1\fs20 @*TSQLRecord@} instances to work with in pure {\i Delphi} code; this is exactly what {\f1\fs20 TSQLRestStorageInMemory} allows, and definitively makes sense for an @*ORM@ framework;
- On the client or server side, you could create calculated fields easily with {\f1\fs20 TSQLRestStorageInMemory} dedicated "getter" methods written in {\i Delphi}, whereas {\i SQlite3} in-memory tables will need additional SQL coding;
- {\i SQLite3} tables are stored in the main database file - in some cases, it could be much convenient to provide some additional table content in some separated database file (for a round robin table, a configuration table written in JSON, some content to be shared among users...): this is made possible using our {\f1\fs20 JSON} or {\f1\fs20 Binary} virtual table modules (but, to be honest, the {\f1\fs20 @*ATTACH DATABASE@} statement could provide a similar feature);
- The {\f1\fs20 TSQLRestStorageInMemory} class can be used stand-alone, i.e. without the {\i SQLite3} engine so it could be used to produce small efficient server software - see the "{\f1\fs20 SQLite3\\Samples\\01 - In Memory ORM}" folder.
:   In-Memory tables
A first way of using @*static@ tables, independently from the {\i SQLite3} engine, is to call the {\f1\fs20 TSQLRestServer. StaticDataCreate} method.
This method is only to be called server-side, of course. For the Client, there is no difference between a regular and a static table.
The in-memory {\f1\fs20 @*TSQLRestStorageInMemory@} instance handling the storage can be accessed later via the {\f1\fs20 StaticDataServer[]} property array of {\f1\fs20 TSQLRestServer}.
As we just stated, this primitive but efficient database engine can be used without need of the {\i SQLite3} database engine to be linked to the executable, saving some KB of code if necessary. It will be enough to handle most basic @*REST@ful requests.
:76   In-Memory virtual tables
A more advanced and powerful way of using @*static@ tables is to define some classes inheriting from {\f1\fs20 @*TSQLRecordVirtualTableAutoID@}, and associate them with some {\f1\fs20 TSQLVirtualTable} classes. The {\f1\fs20 TSQLRecordVirtualTableAutoID} parent class will specify that associated @*virtual table@ modules will behave like normal {\i SQLite3} tables, so will have their {\f1\fs20 RowID} property computed at {\f1\fs20 INSERT}).
For instance, the supplied regression tests define such two tables with three columns, named {\f1\fs20 FirstName}, {\f1\fs20 YearOfBirth} and {\f1\fs20 YearOfDeath}, after the @*published properties@ definition:
!  TSQLRecordDali1 = class(TSQLRecordVirtualTableAutoID)
!  private
!    fYearOfBirth: integer;
!    fFirstName: RawUTF8;
!    fYearOfDeath: word;
!  published
!    property FirstName: RawUTF8 read fFirstName write fFirstName;
!    property YearOfBirth: integer read fYearOfBirth write fYearOfBirth;
!    property YearOfDeath: word read fYearOfDeath write fYearOfDeath;
!  end;
!  TSQLRecordDali2 = class(TSQLRecordDali1);
Both class types are then added to the {\f1\fs20 @*TSQLModel@} instance of the application, common to both Client and Server side:
!  ModelC := TSQLModel.Create(
!    [TSQLRecordPeople,  (...)
!     TSQLRecordDali1,TSQLRecordDali2],'root');
Then, on the Server side, the corresponding Virtual Table modules are associated with those both classes:
!  ModelC.VirtualTableRegister(TSQLRecordDali1,TSQLVirtualTableJSON);
!  ModelC.VirtualTableRegister(TSQLRecordDali2,TSQLVirtualTableBinary);
Thanks to the {\f1\fs20 VirtualTableRegister} calls, on the server side, the '{\f1\fs20 JSON}' and '{\f1\fs20 Binary}' Virtual Table modules will be launched automatically when the {\i @*SQLite3@} DB connection will be initialized:
!  Client := TSQLRestClientDB.Create(ModelC,nil,Demo,TSQLRestServerTest);
This {\f1\fs20 @*TSQLRestClientDB@} has in fact a {\f1\fs20 @*TSQLRestServerDB@} instance running, which will be used for all Database access, including Virtual Table process.
Two files will be created on disk, named '{\f1\fs20 Dali1.json}' and '{\f1\fs20 Dali2.data}'. As stated above, the JSON version will be much bigger, but also more easy to handle from outside the application.
From the code point of view, there is no difference in our @*ORM@ with handling those virtual tables, compared to regular {\f1\fs20 @*TSQLRecord@} tables. For instance, here is some code extracted from the supplied regression tests:
!!  if aClient.TransactionBegin(TSQLRecordDali1) then
!  try
!    // add some items to the file
!    V2.FillPrepare(aClient,'LastName=:("Dali"):');
!    n := 0;
!    while V2.FillOne do begin
!      VD.FirstName := V2.FirstName;
!      VD.YearOfBirth := V2.YearOfBirth;
!      VD.YearOfDeath := V2.YearOfDeath;
!      inc(n);
!!      Check(aClient.Add(VD,true)=n,Msg);
!    end;
!    // update some items in the file
!    for i := 1 to n do begin
!!      Check(aClient.Retrieve(i,VD),Msg);
!      Check(VD.ID=i);
!      Check(IdemPChar(pointer(VD.FirstName),'SALVADOR'));
!      Check(VD.YearOfBirth=1904);
!      Check(VD.YearOfDeath=1989);
!      VD.YearOfBirth := VD.YearOfBirth+i;
!      VD.YearOfDeath := VD.YearOfDeath+i;
!!      Check(aClient.Update(VD),Msg);
!    end;
!    // check SQL requests
!    for i := 1 to n do begin
!!      Check(aClient.Retrieve(i,VD),Msg);
!      Check(VD.YearOfBirth=1904+i);
!      Check(VD.YearOfDeath=1989+i);
!    end;
!!    Check(aClient.TableRowCount(TSQLRecordDali1)=1001);
!!    aClient.Commit;
!  except
!!    aClient.RollBack;
!  end;
A {\f1\fs20 Commit} is needed from the Client side to write anything on disk. From the Server side, in order to create disk content, you'll have to explicitly call such code on purpose:
As we already noticed, data will be written by default on disk with our {\f1\fs20 @*TSQLRestStorageInMemory@}-based virtual tables. In fact, the {\f1\fs20 Commit} method in the above code will call {\f1\fs20 TSQLRestStorageInMemory.UpdateFile}.
Please note that the {\i @*SQlite3@} engine will handle any Virtual Table just like regular {\i SQLite3} tables, concerning the @*atomic@ity of the data. That is, if no explicit @*transaction@ is defined (via {\f1\fs20 TransactionBegin / Commit} methods), such a transaction will be performed for every database modification (i.e. all @*CRUD@ operations, as {\f1\fs20 INSERT / UPDATE / DELETE}). The {\f1\fs20 TSQLRestStorageInMemory. UpdateToFile} method is not immediate, because it will write all table data each time on disk. It is therefore mandatory, for performance reasons, to nest multiple modification to a Virtual Table with such a transaction, for better performance. And in all cases, it is the standard way of using the ORM. If for some reason, you later change your mind and e.g. move your table from the {\f1\fs20 TSQLVirtualTableJSON / TSQLVirtualTableBinary} engine to the default {\i SQlite3} engine, your code could remain untouched.
It is possible to force the In-Memory virtual table data to stay in memory, and the {\f1\fs20 COMMIT} statement to write nothing on disk, using the following property:
! Server.StaticVirtualTable[TSQLRecordDali1].CommitShouldNotUpdateFile := true;
In order to create disk content, you'll then have to explicitly call the corresponding method on purpose:
! Server.StaticVirtualTable[TSQLRecordDali1].UpdateToFile;
Since {\f1\fs20 StaticVirtualTable} property is only available on the Server side, you are the one to blame if your client updates the table data and this update never reaches the disk!
:   In-Memory and ACID
For data stored in memory, the {\f1\fs20 TSQLRestStorageInMemory} table is @*ACID@.\line It means that concurrent access will be consistent and work safely, as expected.
On disk, this kind of table is ACID only when its content is written to the file.\line I mean, the whole file which will be written in an ACID way. The file will always be consistent.
The exact process of these in-memory tables is that each time you write some new data to a {\f1\fs20 TSQLRestStorageInMemory} table:
- It will be ACID in memory (i.e. work safely in concurrent mode);
- Individual writes (INSERT/UPDATE/DELETE) won't automatically be written to file;
- COMMIT will by default write the whole table to file (either as JSON or compressed binary);
- COMMIT won't write the data to file if the {\f1\fs20 CommitShouldNotUpdateFile} property is set to TRUE;
- ROLLBACK process won't do anything, so won't be ACID - but since your code may later use a real RDBMS, it is a good habit to always write the command, like in the sample code above, as {\f1\fs20 except aClient.RollBack}.
When you write the data to file, the whole file is rewritten: it seems not feasible to write the data to disk at every write - in this case, SQLite3 in exclusive mode will be faster, since it will write only the new data, not the whole table content.
This may sound like a limitation, but on our eyes, it could be seen more like a feature. For a particular table, we do not need nor want to have a whole RDBMS/SQL engine, just direct and fast access to a {\f1\fs20 TObjectList}. The feature is to integrate it with our @*REST@ engine, and still be able to store your data in a regular database later ({\i SQLite3} or external), if it appears that {\f1\fs20 TSQLRestStorageInMemory} storage is too limited for your process.
:93  Redirect to an external TSQLRest
Sometimes, having all database process hosted in a single process may not be enough. You can use the {\f1\fs20 TSQLRestServer.RemoteDataCreate()} method to instantiate a {\f1\fs20 TSQLRestStorageRemote} class which will redirect all ORM operation to a specified {\f1\fs20 TSQLRest} instance, may be remote (via {\f1\fs20 TSQLRestClientHttp}) or in-process ({\f1\fs20 TSQLRestServer}). REST @**redirection@ may be enough in simple use cases, when full @147@ could be oversized.
For instance, in {\f1\fs20 TTestExternalDatabase} regression tests, you will find the following code:
!  aExternalClient := TSQLRestClientDB.Create(fExternalModel,nil,'testExternal.db3',TSQLRestServerDB);
!!  historyDB := TSQLRestServerDB.Create(
!!    TSQLModel.Create([TSQLRecordMyHistory],'history'),
!!    'history.db3',false);
!historyDB.Model.Owner := historyDB;
!  historyDB.DB.Synchronous := smOff;
!  historyDB.DB.LockingMode := lmExclusive;
!  historyDB.CreateMissingTables;
!    'history.db3',false);
!!  aExternalClient.Server.RemoteDataCreate(TSQLRecordMyHistory,historyDB);
!  aExternalClient.Server.DB.Synchronous := smOff;
!  aExternalClient.Server.DB.LockingMode := lmExclusive;
!  aExternalClient.Server.CreateMissingTables;
!...
It will create two {\f1\fs20 SQLite3} databases, one main "{\f1\fs20 testExternal.db3}", and a separated "{\f1\fs20 history.db3}" database. Both will use {\i synch off} and {\i lock exclusive} access mode - see @60@ just above.
In the "{\f1\fs20 history.db3}" file, there will be the {\f1\fs20 MyHistory} table, whereas in {\f1\fs20 testExternal.db3}", there won't be any {\f1\fs20 MyHistory} table. All {\f1\fs20 TSQLRecordMyHistory} CRUD process will be transparently redirected to {\f1\fs20 historyDB}.
Then any ORM access from the main {\f1\fs20 aExternalClient} to the {\f1\fs20 TSQLRecordMyHistory} table via will be redirected, via an hidden {\f1\fs20 TSQLRestStorageRemote} instance, to {\f1\fs20 historyDB}. There won't be any noticeable performance penalty - on the contrary a separated database will be much better.
An alternative may have been to use the {\f1\fs20 ATTACH TABLE} statement at {\i SQLite3} level, but it will have been only locally, and you will not be able to switch to another database engine. Whereas the {\f1\fs20 RemoteDataCreate()} method is generic, and will work with external databases - see @27@, even {\i @*NoSQL@} databases - see @84@, or remote {\i mORMot} servers, accessible via a {\f1\fs20 TSQLRestClientHTTP} instance. The only prerequirement is that all {\f1\fs20 TSQLRecord} classes in the main model do exist in the redirected database model.
Note that the redirected {\f1\fs20 TSQLRest} instance can have its own model, its own authentication and authorization scheme, its own caching policy. It may be of great interrest when tuning your application.\line Be aware that if you use {\f1\fs20 @*TRecordReference@} published fields, the model should better be shared among the local and redirected {\f1\fs20 TSQLRest} instances, or at least the {\f1\fs20 TSQLRecord} classes should have the same order - otherwise the {\f1\fs20 TRecordReference} values will point to the wrong table, depending on the side the query is run.
See @%%mORMotDesign3@ and  @%%ArchServerFull@ for some explanation about how this redirection feature interacts with other abilities of the framework.
One practical application of this redirection pattern may be with a typical corporate business.\line There may be a main {\i mORMot} server, at corporation headquarters, then local {\i mORMot} servers at each branch office, hosting applications for end users on the local network:
\graph HostingRedirection Corporate Servers Redirection
subgraph cluster_0 {
label="Main Office";
\Main¤Server\External DB
}
subgraph cluster_1 {
label="           Office A";
\Main¤Server\Local¤Server A\HTTP
\Local¤Server A\Client 1
\Local¤Server A\Client 2
\Local¤Server A\Client 3\local¤network
}
subgraph cluster_2 {
label="          Office B";
\Main¤Server\Local¤Server B\HTTP
\Local¤Server B\Client  1
\Local¤Server B\Client  2
\Local¤Server B\Client  3
\Local¤Server B\Client  4\local¤network
}
\
Each branch office may have its own {\f1\fs20 TSQLRecord} dedicated table, with all its data. Some other tables will be shared among local offices, like global configuration.\line Creating a dedicated table can be done in Delphi code by creating your own class type:
!type
!  TSQLRecordCustomerAbstract = class // never declared in Model
!  .... // here the fields used for Customer business
!  end;
!  TSQLRecordCustomerA = class(TSQLRecordCustomerAbstract); // for office A
!  TSQLRecordCustomerB = class(TSQLRecordCustomerAbstract); // for office B
!
!  TSQLRecordCustomerClass = class of TSQLRecordCustomerAbstract;
Here, {\f1\fs20 TSQLRecordCustomerA} may be part only of the Office A server's {\f1\fs20 TSQLModel}, and {\f1\fs20 TSQLRecordCustomerB} only of the Office B server's {\f1\fs20 TSQLModel}. It will increase security, and, in the main headequarters server, both {\f1\fs20 TSQLRecordCustomerA} and {\f1\fs20 TSQLRecordCustomerB} classes will be part of the {\f1\fs20 TSQLModel}, and dedicated interface-based services will be able to publish some high-level data and statistics about all stored tables.\line Then you can use a {\f1\fs20 TSQLRecordCustomerClass} variable in your client code, which will contain either {\f1\fs20 TSQLRecordCustomerA} or {\f1\fs20 TSQLRecordCustomerB}, depending on the place it runs on, and the server it is connected to.\line On the main server, each office will have its own storage table in the (external) database, named {\f1\fs20 CustomerA} or {\f1\fs20 CustomerB}.
You will benefit of the caching abilities - see @39@ - of each {\f1\fs20 TSQLRest} instance. You may have some cache tuned at a local site, whereas the cache in the main database will remain less aggressive, but safer.
Furthermore, even on a single-siter server, a {\f1\fs20 TSQLRecordHistory} table, or more generaly any aggregation data may benefit to be hosted locally or on cheap storage, whereas the main database will stay on SSD or SAS. Thanks to this redirection feature, you can tune your hosting as expected.
Finally, if your purpose is to redirect all tables of a given {\f1\fs20 TSQLRestServer} to another remote {\f1\fs20 TSQLRestServer} (for security or hosting purpose), you may consider using {\f1\fs20 TSQLRestServerRemoteDB} instead. This class will redirect all tables to one external instance.
Note that both {\f1\fs20 TSQLRestStorageRemote} and {\f1\fs20 TSQLRestServerRemoteDB} classes do not support yet the {\i Virtual Tables} mechanism of {\i SQlite3}. So if you use those features, you may not be able to run JOINed queries from the redirected instance: in fact, the main SQlite3 engine will complain about a missing {\f1\fs20 MyHistory} table in "{\f1\fs20 testExternal.db3}". We will eventually define the needed {\f1\fs20 TSQLVirtualTableRemote} and {\f1\fs20 TSQLVirtualTableCursorRemote} classes to implement this feature.
Sadly, this redirection pattern won't work if the connection is lost. The main office server needs to be always accessible so that the local offices continue to work. You may consider using @147@ to allow the local offices to work with their own local copy of the master data. @*Replication@ sounds in fact preferred than simple redirection, especially in terms of network and resource use, in some cases.
:  Virtual Tables to access external databases
As will be stated @27@, some external databases may be accessed by our ORM.
The @*Virtual Table@ feature of {\i @*SQLite3@} will allow those remote tables to be accessed just like "native" {\i SQLite3} tables - in fact, you may be able e.g. to write a valid SQL query with a {\f1\fs20 @*JOIN@} between {\i SQlite3} tables, {\i @*MS SQL@ Server, @*MySQL@, @*FireBird@, @*PostgreSQL@, @*MySQL@, @*DB2@, @*Informix@} and {\i @*Oracle@} databases, even with multiple connections and several remote servers. Think as an ORM-based {\i Business Intelligence} from any database source. Added to our code-based reporting engine (able to generate @*pdf@), it could be a very powerful way of consolidating any kind of data.
In order to define such {\i external} tables, you define your regular {\f1\fs20 @*TSQLRecord@} classes as usual, then a call to the {\f1\fs20 @**VirtualTableExternalRegister@()}  or {\f1\fs20 @**VirtualTableExternalMap@()} functions will define this class to be managed as a virtual table, from an external database engine. Using a dedicated external database server may allow better response time or additional features (like data sharing with other applications or languages). Server-side may omit a call to {\f1\fs20 VirtualTableExternalRegister()} if the need of an internal database is expected: it will allow custom database configuration at runtime, depending on the customer's expectations (or license).
:  Virtual tables from the client side
For external databases - see @27@ - the SQL conversion will be done on the fly in a more advanced way, so you should be able to work with such virtual tables from the client side without any specific model notification. In this case, you can safely define your tables as {\f1\fs20 TSQLValue1 = class(TSQLRecord)}, with no further code on client side.
When working with {\i static} (in-memory / {\f1\fs20 TObjectList}) storage, if you expect all ORM features to work remotely, you need to notify the Client-side model that a table is implemented as virtual. Otherwise you may encounter some SQL errors when executing requests, like "{\i no such column: ID}".
For instance, imagine you defined two in-memory JSON virtual tables on Server side:
!type
!  TSQLServer = class(TSQLRestServerDB)
!  private
!    FHttpServer: TSQLHttpServer;
!  public
!    constructor Create;
!    destructor Destroy; override;
!  end;
!
!constructor TSQLServer.Create;
!var aModel: TSQLModel;
!begin
!  aModel := CreateModel;
!!  aModel.VirtualTableRegister(TSQLValue1, TSQLVirtualTableJSON);
!!  aModel.VirtualTableRegister(TSQLValue2, TSQLVirtualTableJSON);
!  aModel.Owner := self; // model will be released with TSQLServer instance
!  inherited Create(aModel, ChangeFileExt(ParamStr(0), '.db'), True);
!  Self.CreateMissingTables(0);
!  FHttpServer:= TSQLHttpServer.Create('8080', Self);
!end;
!
!destructor TSQLServer.Destroy;
!begin
!  FHttpServer.Free;
!  inherited;
!end;
You will need to specify also on the client side that those {\f1\fs20 TSQLValue1} and {\f1\fs20 TSQLValue2} tables are virtual.
You have several possibilities:
- Inherit each table not from {\f1\fs20 TSQLRecord}, but from {\f1\fs20 @**TSQLRecordVirtualTableAutoID@}, as was stated above as standard procedure for virtual tables - see @76@;
- If your tables are defined as {\f1\fs20 TSQLRecord}, ensure that the Client side set the table property of its own model to {\f1\fs20 rCustomAutoID};
- If your tables are defined as {\f1\fs20 TSQLRecord}, ensure that both Client and Server set the table property of its own model to {\f1\fs20 rCustomAutoID}.
First option could be done as such:
!type
!!  TSQLValue1 = class(TSQLRecordVirtualTableAutoID)
!  (...)
!!  TSQLValue2 = class(TSQLRecordVirtualTableAutoID)
!  (...)
Or, in case the table is defined as {\f1\fs20 TSQLValue1 = class(TSQLRecord)}, the client model could be updated as such:
!type
!  TSQLClient = class(TSQLHttpClient)
!  public
!    constructor Create;
!  end;
!
!constructor TSQLClient.Create;
!var aModel: TSQLModel;
!begin
!  aModel:= CreateModel;
!!  aModel.Props[TSQLValue1].Kind := rCustomAutoID;
!!  aModel.Props[TSQLValue2].Kind := rCustomAutoID;
!  aModel.Owner := self; // model will be released within TSQLServer instance
!  inherited Create('127.0.0.1', '8080', aModel);
!  SetUser('Admin', 'synopse');
!end;
Or, in case the table is defined as {\f1\fs20 TSQLValue1 = class(TSQLRecord)}, perhaps the easiest way of doing it, is to set the property when creating the shared model:
!function CreateModel: TSQLModel;
!begin
!  result:= TSQLModel.Create([TSQLAuthGroup, TSQLAuthUser, TSQLValue1, TSQLValue2]);
!!  result.Props[TSQLValue1].Kind := rCustomAutoID;
!!  result.Props[TSQLValue2].Kind := rCustomAutoID;
!end;
The easiest is definitively to let your static in-memory tables inherit from {\f1\fs20 TSQLRecordVirtualTableAutoID}. Just use the framework by the book - see @76@.
Once again, this restriction does not apply to @27@.
:27External SQL database access
%cartoon05.png
Our @*ORM@ @*REST@ful framework is able to access most available database engines, via a set of generic units and classes. Both @*SQL@ and @*NoSQL@ engines could be accessed - quite a unique feature in the ORM landscape (in {\i Delphi}, of course, but also in Java or C# environments).
Remember the diagram introducing {\i mORMot}'s @42@:
%%mORMotDBDesign
\page
The framework still relies on {\i @*SQLite3@} as its SQL core on the server, but a dedicated mechanism allows access to any remote database, and mixes those tables content with the native ORM tables of the framework. Thanks to the unique @20@ mechanism of {\i SQLite3}, those external tables may be accessed as native {\i SQLite3} tables in our SQL statements, even for NoSQL engines.
|%10%90
|\b Mode|Engines\b0
|SQL|{\i @*SQLite3@, @*Oracle@, @*NexusDB@, @*MS SQL@, @*Jet/MSAccess@, @*FireBird@, @*MySQL@, @*PostgreSQL@, IBM @*DB2@, IBM @*Informix@}\line See @126@
|NoSQL|{\i @*MongoDB@}, {\f1\fs20 TObjectList} with JSON or binary disk persistence\line See @83@ and @57@
|%
You can even mix databases, i.e. the same {\i mORMot} ORM could persist, at the same time, its data in several databases, some {\f1\fs20 TSQLRecord} as fast internal {\i SQLite3} tables or as {\f1\fs20 TObjectList}, others in a {\i PostgreSQL} database (tied to an external reporting/SAP engine), and e.g. flat consolidated data in a {\i MongoDB} instance.
\page
:126 SynDB direct RDBMS access
External {\i Relational Database Management System} (@*RDBMS@) can be accessed via our {\f1\fs20 @*SynDB@.pas} units. Then, the framework ORM is able to access them via the {\f1\fs20 mORMotDB.pas} bridge unit. But you can use the {\f1\fs20 SynDB.pas} units directly, without any link to our ORM.
The current list of handled data access libraries is:
|%13%25%62
|\b Provider|{\f1\fs20 SynDB} Unit|RDBMS Engines\b0
|{\i SQLite3}|{\f1\fs20 SynDBSQLite3.pas}|direct {\i SQLite3} access (as dll or linked to the exe)\line See @127@
|{\i Oracle}|{\f1\fs20 SynDBOracle.pas}|direct {\i Oracle} access (via OCI)\line See @117@
|{\i @*OleDB@}|{\f1\fs20 SynOleDB.pas}|{\i MS SQL, Jet/MSAccess} or others\line See @118@
|{\i @*ODBC@}|{\f1\fs20 SynDBODBC.pas}|{\i MS SQL, FireBird, MySQL, PostgreSQL, IBM DB2, Informix} or others\line See @118@
|{\i @*Zeos@Lib}|{\f1\fs20 SynDBZeos.pas}|{\i MS SQL, SQLite3, FireBird, MySQL, PostgreSQL}\line See @94@
|{\f1\fs20 DB.pas}/\line {\f1\fs20 @**TDataset@}|{\f1\fs20 SynDBDataset.pas}|@*NexusDB@ and databases supported by @*DBExpress@, @*FireDAC@, @*AnyDac@, @*UniDAC@, @*BDE@...\line See @119@
|HTTP|{\f1\fs20 SynDBRemote.pas}|remote access to any {\f1\fs20 SynDB} database, over HTTP
|%
This list is not closed, and may be completed in the near future. Any help is welcome here: it is not difficult to implement a new unit, following the patterns already existing. You may start from an existing driver (e.g. {\i Zeos} or {\i Alcinoe} libraries). Open Source contribution are always welcome!
Thanks to the design of our {\f1\fs20 SynDB.pas} classes, it was very easy (and convenient) to implement {\i SQLite3} direct access. It is even used for our regression tests, in order to implement stand-alone unitary testing.
An {\i Oracle} dedicated direct access was added, because all available OleDB providers for Oracle (i.e. both Microsoft's and Oracle's) do have problems with handling BLOB, and we wanted our Clients to have a light-weight and as fast as possible access to this great database.
In fact, {\i OleDB} is a good candidate for database access with good performance, Unicode native, with a lot of available providers. Thanks to {\i OleDB}, we are already able to access to almost any existing database. The code overhead in the server executable will also be much less than with adding any other third-party {\i Delphi} library. And we will let Microsoft or the {\i OleDB} provider perform all the testing and debugging for each driver.
Since revision 1.17, direct access to the {\i ODBC} layer has been included to the framework database units. It has a wider range of free providers (including e.g. {\i MySQL} or {\i FireBird}), and is the official replacement for {\i OleDB} (next version of {\i MS SQL Server} will provide only ODBC providers, as far as {\i Microsoft} warned its customers).
Since revision 1.18, any {\i ZeosLib} / {\i ZDBC} driver can be used, with fast direct access to the underlying RDBMS client library. Since the {\i ZDBC} library does not rely on {\f1\fs20 DB.pas}, and by-passes the slow {\f1\fs20 TDataSet} component, its performance is very high. The {\i ZDBC} maintainers did a lot of optimizations, especially to work with {\i mORMot}, and this library is a first-class citizen to work with our framework.
Since the same 1.18 revision, {\f1\fs20 DB.pas} can be used with our {\f1\fs20 SynDB.pas} classes. Of course, using {\f1\fs20 TDataset} as intermediate layer will be slower than the {\f1\fs20 SynDB.pas} direct access pattern. But it will allow you to re-use any existing (third-party) database connection driver, which could make sense in case of evolution of an existing application, or to use an unsupported database engine.
Last but not least, the {\f1\fs20 SynDBRemote.pas} unit allows you to  create database applications that perform SQL operations on a remote {\f1\fs20 SynDB} HTTP server, instead of a database server. You can create connections just like any other {\f1\fs20 SynDB} database, but the transmission will take place over HTTP, with no need to install a database client with your application - see @131@.
The following connections are therefore possible:
\graph SynDBLayers SynDB Architecture
\SynDB\Zeos
=Zeos=ZDBC¤(Zeos)
\SynDB\ODBC
\SynDB\OleDB
\SynDB\Oracle
\SynDB\SQLite3
\SynDB\DB
\SynDB\SynDB¤Remote
\SynDB¤Remote\any RDBMS
=DB=DB.pas¤TDataset
\Zeos\Oracle
\Zeos\MS SQL
\Zeos\MySQL
\Zeos\PostgreSQL
\Zeos\Interbase
\Zeos\Firebird
\Zeos\Sybase
\Zeos\SQLite3
\ODBC\Oracle
\ODBC\MS SQL
\ODBC\MySQL
\ODBC\DB2
\ODBC\PostgreSQL
\ODBC\Interbase
\ODBC\Informix
\ODBC\Firebird
\ODBC\Sybase
\ODBC\Jet/Access
\ODBC\Advantage
\OleDB\Oracle
\OleDB\MS SQL
\OleDB\Jet/Access
\OleDB\MySQL
\OleDB\PostgreSQL
\OleDB\Interbase
\OleDB\Firebird
\OleDB\Informix
\OleDB\Sybase
\DB\NexusDB
\DB\AnyDAC
\DB\UniDAC
\DB\DBExpress
\DB\BDE
\UniDAC\Oracle
\UniDAC\ODBC
\UniDAC\MySQL
\UniDAC\DB2
\UniDAC\PostgreSQL
\UniDAC\Interbase
\UniDAC\Firebird
\UniDAC\NexusDB
\UniDAC\SQLite3
\AnyDAC\Oracle
\AnyDAC\MS SQL
\AnyDAC\MySQL
\AnyDAC\PostgreSQL
\AnyDAC\DB2
\AnyDAC\Interbase
\AnyDAC\Firebird
\AnyDAC\ODBC
\AnyDAC\SQLite3
\AnyDAC\Sybase
=AnyDAC=FireDAC¤AnyDAC
\DBExpress\Oracle
\DBExpress\DB2
\DBExpress\MS SQL
\DBExpress\MySQL
\DBExpress\Interbase
\DBExpress\Firebird
\DBExpress\ODBC
\BDE\Paradox
\BDE\Oracle
\BDE\DB2
\BDE\Interbase
\BDE\Informix
\BDE\ODBC
\NexusDB=Oracle=Sybase=Advantage=Paradox=Informix
\AnyDAC=UniDAC=BDE=DBExpress
\
This diagram is a bit difficult to follow at the latest level - but you got the general layered design, I guess. It will be split into smaller focused diagrams later.
:  Direct access to any RDBMS engine
The {\f1\fs20 @**SynDB@.pas} units have the following features:
- Direct fast access via {\i @*OleDB@, @*ODBC@, @*ZDBC@, @*Oracle@} (OCI) or {\i @*SQLite3@} (statically linked or via external {\f1\fs20 dll});
- Thin wrapper around any {\f1\fs20 DB.pas} / {\f1\fs20 TDataset} based components (e.g. @*NexusDB@, @*DBExpress@, @*FireDAC@, @*AnyDAC@, @*UniDAC@, @*BDE@...);
- Generic abstract @*OOP@ layout, with a restricted set of data types, but able to work with any SQL-based database engine;
- Tested with {\i @*MS SQL@ Server 2008/2012, @*Firebird@ 2.5.1, @PostgreSQL@ 9.2/9.3, @*MySQL@ 5.6, IBM @*DB2@ 10.5, Oracle 11g}, and the latest {\i SQLite3} engine;
- Could access any local or remote Database, from any edition of {\i Delphi} (even {\i Delphi 7 personal}, the {\i Turbo Explorer} or {\i Starter edition}), just for free (in fact, it does not use the {\f1\fs20 DB.pas} standard unit and all its dependencies);
- Unicode, even with pre-Unicode version of {\i Delphi} (like {\i Delphi} 7 or 2007), since it uses internally @*UTF-8@ encoding;
- Handle NULL or BLOB content for parameters and results, including stored procedures;
- Avoid most memory copy or unnecessary allocation: we tried to access the data directly from the retrieved data buffer, just as given from {\i OleDB / ODBC} or the low-level database client (e.g. OCI for Oracle, or the {\i SQLite3} engine);
- Designed to achieve the best possible performance on 32-bit or @*64-bit@ Windows: most time is spent in the database provider (OleDB, ODBC, OCI, {\i SQLite3}) - the code layer added to the database client is very thin and optimized;
- Could be safely used in a multi-threaded application/server (with dedicated thread-safe methods, usable even if the database client is not officially multi-thread);
- Allow parameter bindings of @*prepared@ requests, with fast access to any parameter or column name (thanks to {\f1\fs20 @*TDynArrayHashed@});
- Column values accessible with most {\i Delphi} types, including {\f1\fs20 Variant} or generic {\f1\fs20 string / @*WideString@};
- Available {\f1\fs20 ISQLDBRows} interface - to avoid typing {\f1\fs20 try...finally Query.Free end;} and allow one-line SQL statement;
- @*Late-binding@ column access, via a custom variant type when accessing the result sets;
- two kind of optimized {\f1\fs20 @*TDataSet@} result sets: one read-write based on {\f1\fs20 @*TClientDataSet@}, and a much faster read-only {\f1\fs20 @*TSynSQLStatementDataSet@}
- Direct UTF-8 @*JSON@ content creation, with no temporary data copy nor allocation (this feature will be the most used in our JSON-based ORM server);
- High-level catalog / database layout abstract methods, able to retrieve the table and column properties (including indexes), for database reverse-engineering; provide also SQL statements to create a table or an index in a database-abstract manner; those features will be used directly by our ORM;
- Designed to be used with our ORM, but could be used stand-alone (a full {\i Delphi} 7 client executable is just about 200 KB), or even in any existing {\i Delphi} application, thanks to a {\f1\fs20 @*TQuery@}-like wrapper;
- {\f1\fs20 TQuery} {\i emulation class}, for direct re-use with existing code, in replacement to {\f1\fs20 DB.pas} based code (including the deprecated @*BDE@ technology), with huge speed improvement for result sets (since we bypass the slow {\f1\fs20 TDataSet} component);
- Fast and safe remote access over HTTP to any {\f1\fs20 SynDB} engine, without the need to deploy the RDBMS client library with the application;
- Free {\f1\fs20 @**SynDBExplorer@} tool provided, which is a small but efficient way of running queries in a simple User Interface, on all supported engines, and publish as server or consume as client {\f1\fs20 SynDB} remote access over HTTP - see @131@; it is also a good sample program of a stand-alone usage of those libraries.
:  Data types
Of course, our ORM does not need a whole feature set (do not expect to use this database classes with your VCL DB RAD components), but handles directly the basic SQL column types, as needed by our ORM (derived from SQLite's internal column types): {\f1\fs20 NULL, Int64, Double, @*Currency@, DateTime, @*RawUTF8@} and {\f1\fs20 BLOB}.
They are defined as such in {\f1\fs20 @*SynDB@.pas}:
!  TSQLDBFieldType =
!    (ftUnknown, ftNull, ftInt64, ftDouble, ftCurrency, ftDate, ftUTF8, ftBlob);
|%25%65
|\b {\f1\fs20 TSQLDBFieldType}|Content\b0
|{\f1\fs20 ftNull}|Maps the SQL {\f1\fs20 NULL} value
|{\f1\fs20 ftInt64}|Any {\i integer} value, with 64-bit resolution
|{\f1\fs20 ftDouble}|Any {\i floating-point} value, with 64-bit ({\f1\fs20 @*double@}) resolution
|{\f1\fs20 ftCurrency}|Fixed {\i financial} type, with up to 4 fixed decimal digits ({\f1\fs20 @*currency@})
|{\f1\fs20 ftDate}|Date and time, mapping the Delphi {\f1\fs20 TDateTime} type
|{\f1\fs20 ftUTF8}|Unicode text, encoded as @*UTF-8@, with or without size limit
|{\f1\fs20 ftBlob}|Binary content, stored as a {\f1\fs20 RawByteString}
|%
Those types will map low-level database-level access types, not high-level {\i Delphi} types as {\f1\fs20 TSQLFieldType} defined in {\f1\fs20 mORMot.pas}, or the generic huge {\f1\fs20 TFieldType} as defined in the standard VCL {\f1\fs20 DB.pas} unit. In fact, it is more tied to the standard {\i @*SQLite3@} generic types, i.e. NULL, INTEGER, REAL, TEXT, BLOB (with the addition of a {\f1\fs20 ftCurrency} and {\f1\fs20 ftDate} type, for better support of most DB engines) see @http://www.sqlite.org/datatype3.html
You can note that the only {\f1\fs20 string} type handled here uses UTF-8 encoding (implemented using our {\f1\fs20 RawUTF8} type), for cross-{\i Delphi} true Unicode process. Code can access to the textual data via {\f1\fs20 variant, string} or {\f1\fs20 widestring} variables and parameters, but our units will use UTF-8 encoding internally - see @32@. It will therefore interface directly with our ORM, which uses the same encoding. Of course, if the column was not defined as Unicode text in the database, any needed conversion to/from the corresponding charset will take place at the data provider level; but in your user code, you will have always access to the Unicode content.
BLOB columns or parameters are accessed as {\f1\fs20 @*RawByteString@} variables, which may be mapped to a standard {\f1\fs20 TStream} via our {\f1\fs20 TRawByteStringStream}.
:  Database types
In addition to raw data access, the {\f1\fs20 SynDB.pas} unit handles some SQL-level generation, which will be used by our @3@ kernel.
The following RDBMS database engines are defined as such in {\f1\fs20 SynDB.pas}:
! TSQLDBDefinition = (dUnknown, dDefault, dOracle, dMSSQL, dJet,
!   dMySQL, dSQLite, dFirebird, dNexusDB, dPostgreSQL, dDB2, dInformix);
|%25%75
|\b {\f1\fs20 TSQLDBDefinition}|RDBMS tested\b0
|{\f1\fs20 dDefault}|Any database, following the SQL-92 standard
|{\f1\fs20 dOracle}|@*Oracle@ 11g
|{\f1\fs20 dMSSQL}|@*MS SQL@ Server 2008/2012
|{\f1\fs20 dJet}|@*Jet/MSAccess@ (under {\i Win32} only)
|{\f1\fs20 dMySQL}|@*MySQL@ 5.6
|{\f1\fs20 dSQLite}|@SQLite3@ 3.7.11 and up (we supply the latest version for static linking)
|{\f1\fs20 dFirebird}|@*Firebird@ 2.5.1
|{\f1\fs20 dNexusDB}|@*NexusDB@ 3.11
|{\f1\fs20 dPostgreSQL}|@PostgreSQL@ 9.2/9.3
|{\f1\fs20 dDB2}|IBM @*DB2@ 10.5
|{\f1\fs20 dInformix}|IBM @*Informix@ 11.70
|%
The above versions have been tested, but newer or older revisions may also work. Your feedback is welcome: we cannot achieve to test all possible combinations of databases and clients on our own!
The {\f1\fs20 SynDB.pas} unit is able to generate the SQL statements of those engines, for a {\f1\fs20 CREATE TABLE} / {\f1\fs20 CREATE INDEX} command, retrieve metadata (e.g. the tables and fields information), compute the right limit/offset syntax for a {\f1\fs20 SELECT}, compute multi-{\f1\fs20 INSERT} statements - see @99@, check the SQL keywords, define specific schema/owner naming conventions, process date and time values, handle errors and exceptions, or even create a database.
:  SynDB Units
Here are the units implementing the external database-agnostic features:
|%30%70
|\b File|Description\b0
|{\f1\fs20 SynDB.pas}|abstract database direct access classes
|{\f1\fs20 SynOleDB.pas}|@*OleDB@ direct access classes
|{\f1\fs20 SynDBODBC.pas}|@*ODBC@ direct access classes
|{\f1\fs20 SynDBZeos.pas}|@*ZDBC@ direct access classes
|{\f1\fs20 SynDBOracle.pas}|@*Oracle@ DB direct access classes (via OCI)
|{\f1\fs20 SynDBSQLite3.pas}|@*SQLite3@ direct access classes
|{\f1\fs20 SynDBDataset.pas}\line {\f1\fs20 SynDBFireDAC.pas}\line {\f1\fs20 SynDBUniDAC.pas}\line {\f1\fs20 SynDBNexusDB.pas}\line {\f1\fs20 SynDBBDE.pas}|{\f1\fs20 @*TDataset@} ({\f1\fs20 DB.pas}) access classes
|{\f1\fs20 SynDBRemote.pas}|remote access over HTTP
|{\f1\fs20 SynDBVCL}|read/only {\f1\fs20 @*TSynSQLStatementDataSet@} result sets
|{\f1\fs20 SynDBMidasVCL}|read/write {\f1\fs20 @*TClientDataSet@} result sets
|%
It is worth noting that those units only depend on {\f1\fs20 SynCommons.pas}, therefore are independent of the ORM part of our framework (even the remote access). They may be used separately, accessing all those external databases with regular SQL code. Since all their classes inherit from abstract classes defined in {\f1\fs20 SynDB.pas}, switching from one database engine to another (even a remote HTTP access) is just a matter of changing one class type.
:  SynDB Classes
The data is accessed via three families of classes:
- {\i Connection properties}, which store the database high-level properties (like database implementation classes, server and database name, user name and password);
- {\i Connections}, which implements an actual connection to a remote database, according to the specified {\i Connection properties} - of course, there can be multiple {\i connections} for the same {\i connection properties} instance;
- {\i Statements}, which are individual SQL queries or requests, which may be multiple for one existing {\i connection}.
In practice, you define a {\f1\fs20 TSQLDBConnectionProperties} instance, then you derivate {\f1\fs20 TSQLDBConnection} and {\f1\fs20 TSQLDBStatement} instances using dedicated {\f1\fs20 NewConnection} / {\f1\fs20 ThreadSafeConnection} / {\f1\fs20 NewStatement} methods.
Here is the general class hierarchy, for all available remote {\i connection properties}:
\graph HierTSQLDBConnectionProperties TSQLDBConnectionProperties classes hierarchy
rankdir=LR;
\TSQLDBSQLite3ConnectionProperties\TSQLDBConnectionProperties
\TSQLDBZEOSConnectionProperties\TSQLDBConnectionPropertiesThreadSafe
\TSQLDBOracleConnectionProperties\TSQLDBConnectionPropertiesThreadSafe
\TSQLDBFireDACConnectionProperties\TSQLDBDatasetConnectionProperties
\TSQLDBUniDACConnectionProperties\TSQLDBDatasetConnectionProperties
\TSQLDBBDEConnectionProperties\TSQLDBDatasetConnectionProperties
\TSQLDBDatasetConnectionProperties\TSQLDBConnectionPropertiesThreadSafe
\TOleDBMSOracleConnectionProperties\TOleDBOracleConnectionProperties
\TOleDBOracleConnectionProperties\TOleDBConnectionProperties
\TOleDBODBCSQLConnectionProperties\TOleDBConnectionProperties
\TOleDBMySQLConnectionProperties\TOleDBConnectionProperties
\TOleDBMSSQL2012ConnectionProperties\TOleDBMSSQLConnectionProperties
\TOleDBMSSQL2005ConnectionProperties\TOleDBMSSQLConnectionProperties
\TOleDBMSSQLConnectionProperties\TOleDBConnectionProperties
\TOleDBJetConnectionProperties\TOleDBConnectionProperties
\TOleDBAS400ConnectionProperties\TOleDBConnectionProperties
\TOleDBConnectionProperties\TSQLDBConnectionPropertiesThreadSafe
\TODBCConnectionProperties\TSQLDBConnectionPropertiesThreadSafe
\TSQLDBNexusDBConnectionProperties\TSQLDBConnectionPropertiesThreadSafe
\TSQLDBConnectionPropertiesThreadSafe\TSQLDBConnectionProperties
\TSQLDBHTTPConnectionPropertiesAbstract\TSQLDBConnectionProperties
\TSQLDBSocketConnectionProperties\TSQLDBHTTPConnectionPropertiesAbstract
\TSQLDBHttpRequestConnectionProperties\TSQLDBHTTPConnectionPropertiesAbstract
\TSQLDBCurlConnectionProperties\TSQLDBHttpRequestConnectionProperties
\TSQLDBWinHTTPConnectionProperties\TSQLDBHttpRequestConnectionProperties
\TSQLDBWinINetConnectionProperties\TSQLDBHttpRequestConnectionProperties
\
Those classes are the root classes of the {\f1\fs20 SynDB.pas} units, by which most of your database process will be implemented. For instance, the {\i mORMot} framework @*ORM@ only needs a given {\f1\fs20 TSQLDBConnectionProperties} instance to access any external database.
Then the following {\i connection} classes are defined:
\graph HierTSQLDBSQLite3Connection TSQLDBSQLite3Connection classes hierarchy
rankdir=LR;
\TSQLDBConnectionThreadSafe\TSQLDBConnection
\TODBCConnection\TSQLDBConnectionThreadSafe
\TOleDBConnection\TSQLDBConnectionThreadSafe
\TSQLDBBDEConnection\TSQLDBConnectionThreadSafe
\TSQLDBOracleConnection\TSQLDBConnectionThreadSafe
\TSQLDBFireDACConnection\TSQLDBConnectionThreadSafe
\TSQLDBUniDACConnection\TSQLDBConnectionThreadSafe
\TSQLDBZEOSConnection\TSQLDBConnectionThreadSafe
\TSQLDBNexusDBConnection\TSQLDBConnectionThreadSafe
\TSQLDBSQLite3Connection\TSQLDBConnection
\TSQLDBProxyConnection\TSQLDBConnection
\
Each connection may create a corresponding {\i statement} instance:
\graph HierTOleDBStatement TOleDBStatement classes hierarchy
\TSQLDBZEOSStatement\TSQLDBStatementWithParamsAndColumns
\TSQLDBOracleStatement\TSQLDBStatementWithParamsAndColumns
\TSQLDBDatasetStatementAbstract\TSQLDBDatasetStatement
\TSQLDBUniDACStatement\TSQLDBDatasetStatement
\TSQLDBFireDACStatement\TSQLDBDatasetStatementAbstract
\TSQLDBBDEStatement\TSQLDBDatasetStatement
\TSQLDBNexusDBStatement\TSQLDBDatasetStatement
\TSQLDBDatasetStatement\TSQLDBStatementWithParamsAndColumns
\TODBCStatement\TSQLDBStatementWithParamsAndColumns
\TSQLDBStatementWithParamsAndColumns\TSQLDBStatementWithParams
\TSQLDBStatementWithParams\TSQLDBStatement
\TSQLDBSQLite3Statement\TSQLDBStatement
\TOleDBStatement\TSQLDBStatement
\TSQLDBProxyStatement\TSQLDBStatement
\
In the above hierarchy, {\f1\fs20 TSQLDBDatasetStatementAbstract} is used to allow the use of custom classes for parameter process, e.g. {\f1\fs20 TADParams} for @*FireDAC@ (which features {\i Array DML}).
Some dedicated {\f1\fs20 Exception} classes are also defined:
\graph HierESQLQueryException ESQLQueryException classes hierarchy
\ESQLDBDataset\Exception
\ESQLDBBDE\ESQLDBDataset
\ESQLDBFireDAC\ESQLDBDataset
\ESQLDBUniDAC\ESQLDBDataset
\ESQLDBException\Exception
\EODBCException\ESQLDBException
\EOleDBException\ESQLDBException
\ESQLDBZEOS\ESQLDBException
\ESQLDBRemote\ESQLDBException
\ESQLDBOracle\Exception
\ESQLQueryException\Exception
\
Check the {\f1\fs20 TestOleDB.dpr} sample program, located in {\f1\fs20 SQlite3} folder, using our {\f1\fs20 SynOleDB} unit to connect to a local {\i @*MS SQL@ Server 2008 R2 Express edition}, which will write a file with the JSON representation of the {\f1\fs20 Person.Address} table of the sample database {\i AdventureWorks2008R2}.
:137  ISQLDBRows interface
The easiest is to stay at the {\f1\fs20 TSQLDBConnectionProperties} level, using the {\f1\fs20 Execute()} methods of this instance, and access any returned data via an {\f1\fs20 @**ISQLDBRows@} interface. It will automatically use a thread-safe connection to the database, in an abstracted way.
Typical use of {\f1\fs20 SynDB.pas} classes could be:
- Initialize a shared {\f1\fs20 TSQLDBConnectionProperties} instance;
- Execute statements directly from this instance's {\f1\fs20 Execute*()} methods.
Defining a database connection is as easy as:
!var Props: TSQLDBConnectionProperties;
! ...
!  Props := TOleDBMSSQLConnectionProperties.Create('.\\SQLEXPRESS','AdventureWorks2008R2','','');
!  try
!    UseProps(Props);
!  finally
!    Props.Free;
!  end;
Depending on the {\f1\fs20 TSQLDBConnectionProperties} sub-class, input parameters do vary. Please refer to the documentation of each {\f1\fs20 Create() constructor} to set all parameters as expected.
Then any sub-code is able to execute any SQL request, with optional bound parameters, as such:
!procedure UseProps(Props: TSQLDBConnectionProperties);
!var I: ISQLDBRows;
!begin
!  I := Props.Execute('select * from Sales.Customer where AccountNumber like ?',['AW000001%']);
!  while I.Step do
!    assert(Copy(I['AccountNumber'],1,8)='AW000001');
!end;
In this procedure, no {\f1\fs20 TSQLDBStatement} is defined, and there is no need to add a {\f1\fs20 try ... finally Query.Free; end;} block.
In fact, the {\f1\fs20 MyConnProps.Execute} method returns a {\f1\fs20 TSQLDBStatement} instance as a {\f1\fs20 ISQLDBRows}, which methods can be used to loop for each result row, and retrieve individual column values. In the code above, {\f1\fs20 I['FirstName']} will in fact call the {\f1\fs20 I.Column[]} default property, which will return the column value as a {\f1\fs20 variant}. You have other dedicated methods, like {\f1\fs20 ColumnUTF8} or {\f1\fs20 ColumnInt}, able to retrieve directly the expected data.
Note that all bound parameters will appear within the SQL statement, when @*log@ged using our {\f1\fs20 TSynLog} classes - see @73@.
:195  Using properly the ISQLDBRows interface
You may have noticed in the previous code sample, that we used a {\f1\fs20 UseProps()} sub-procedure. This was made on purpose.
We may have written our little test as such:
!var Props: TSQLDBConnectionProperties;
!!    I: ISQLDBRows;
! ...
!  Props := TOleDBMSSQLConnectionProperties.Create('.\\SQLEXPRESS','AdventureWorks2008R2','','');
!  try
!!    I := Props.Execute('select * from Sales.Customer where AccountNumber like ?',['AW000001%']);
!!    while I.Step do
!!      assert(Copy(I['AccountNumber'],1,8)='AW000001');
!  finally
!    Props.Free;
!  end;
!end;
In fact, you should {\b not} use this pattern. This code will lead to an unexpected {\i access violation} at runtime.
Behind the scene, as will be detailed @46@, the compiler is generating some hidden code to finalize the {\f1\fs20 I: ISQLDBRows} local variable, as such:
! ...
!  finally
!    Props.Free;
!  end;
!!  I := nil; // this is generated by the compiler, just before the final "end;"
!end;
So {\f1\fs20 ISQLDBRows} is released {\i after} the {\f1\fs20 Props} instance, and an access violation occurs.
The correct way to write is either to use a sub-function (which will release the local {\f1\fs20 ISQLDBRows} when the function leaves), or explicitely release the interface variable:
!    while I.Step do
!      assert(Copy(I['AccountNumber'],1,8)='AW000001');
!  finally
!!    I := nil; // release local variable
!    Props.Free;
!  end;
Of course, most of the time you will initialize your {\f1\fs20 TSQLDBConnectionProperties} globally for your process, then release it when it ends. Each request will take place in its own sub-method, so will be released before the main {\f1\fs20 TSQLDBConnectionProperties} instance if freed.
Last but not least, it is worth writing that you should {\i not} create a {\f1\fs20 TSQLDBConnectionProperties} instance each time you need to access the database, since you will probably lose most of the {\f1\fs20 SynDB} features, like a per-thread connection pool, or statement cache.
:136  Late-binding
We implemented @*late-binding@ access of column values, via a custom variant time. It uses the internal mechanism used for {\i Ole Automation}, here to access column content as if column names where native object properties.
The resulting {\i Delphi} code to write is just clear and obvious:
!procedure UseProps(Props: TSQLDBConnectionProperties);
!var Row: Variant;
!begin
!  with Props.Execute('select * from Sales.Customer where AccountNumber like ?',
!    ['AW000001%'],@Row) do
!    while Step do
!      assert(Copy(Row.AccountNumber,1,8)='AW000001');
!end;
Note that {\f1\fs20 Props.Execute} returns an {\f1\fs20 ISQLDBRows} interface, so the code above will initialize (or reuse an existing) thread-safe connection (OleDB uses a per-thread model), initialize a statement, execute it, access the rows via the {\f1\fs20 Step} method and the {\f1\fs20 Row} variant, retrieving the column value via a direct {\f1\fs20 Row.AccountNumber} statement.
The above code is perfectly safe, and all memory will be released with the reference count garbage-collector feature of the {\f1\fs20 ISQLDBRows} interface. You are not required to add any {\f1\fs20 try..finally Free; end} statements in your code.
This is the magic of late-binding in {\i Delphi}. Note that a similar feature is available for our {\f1\fs20 SynBigTable} unit.
In practice, this code is slower than using a standard property based access, like this:
!while Step do
!  assert(Copy(ColumnUTF8('AccountNumber'),1,8)='AW000001');
But the first version, using late-binding of column name, just sounds more natural.
Of course, since it is {\i late}-binding, we are not able to let the compiler check at compile time for the column name. If the column name in the source code is wrong, an error will be triggered at runtime only.
First of all, let's see the fastest way of accessing the row content.
In all cases, using the textual version of the column name ({\f1\fs20 'AccountNumber'}) is slower than using directly the column index. Even if our {\f1\fs20 @*SynDB@.pas} library uses a fast lookup using hashing, the following code will always be faster:
!var Customer: Integer;
!begin
!  with Props.Execute(
!    'select * from Sales.Customer where AccountNumber like ?',
!    ['AW000001%'],@Customer) do begin
!    Customer := ColumnIndex('AccountNumber');
!    while Step do
!      assert(Copy(ColumnString(Customer),1,8)='AW000001');
!  end;
!end;
But to be honest, after profiling, most of the time is spend in the {\f1\fs20 Step} method, especially in {\f1\fs20 fRowSet.GetData}. In practice, I was not able to notice any speed increase worth mentioning, with the code above.
Our name lookup via a hashing function (i.e. {\f1\fs20 TDynArrayHashed}) just does its purpose very well.
On the contrary the {\i Ole-Automation} based late-binding was found out to be slower, after profiling. In fact, the {\f1\fs20 Row.AccountNumber} expression calls an hidden {\f1\fs20 DispInvoke} function, which is slow when called multiple times. Our {\f1\fs20 SynCommons.pas} unit is able to hack the VCL, and by patching the VCL code in-memory, will call an optimized version of this function. Resulting speed is very close to direct {\f1\fs20 Column['AccountNumber']} call. See @SDD-DI-2.2.3@.
:134  TDataset and SynDB
Since our {\f1\fs20 SynDB.pas} unit does not rely on the Delphi's {\f1\fs20 DB.pas} unit, its result sets do not inherit from the {\f1\fs20 @**TDataset@}.
As a benefit, those result sets will be much faster, when accessed from your object code.\line But as a drawback, you won't be able to use them in your regular VCL applications.
In order to easily use the {\f1\fs20 SynDB.pas} unit with VCL components, you can create {\f1\fs20 TDataSet} results sets from any {\f1\fs20 SynDB} query.\line You have access to two kind of optimized {\f1\fs20 @*TDataSet@} result sets:
|%30%15%25%30
|\b TDataSet class|Operation|Unit|Remark\b0
|{\f1\fs20 @*TClientDataSet@}|read write|{\f1\fs20 SynDBMidasVCL.pas}|Slow due to memory copy
|{\f1\fs20 @*TSynSQLStatementDataSet@}|read only|{\f1\fs20 SynDBVCL.pas}|Fast due to direct mapping
|%
You can therefore assign the result of a {\f1\fs20 SynDB} request to any {\f1\fs20 TDataSource}, as such for our fast {\f1\fs20 TSynSQLStatementDataSet} read/only storage:
!  ds1.DataSet.Free; // release previous TDataSet
!  ds1.DataSet := ToDataSet(ds1,aProps.Execute('select * from people',[]));
or for a {\f1\fs20 TClientDataSet} kind of in-memory storage:
!  ds1.DataSet.Free; // release previous TDataSet
!  ds1.DataSet := ToClientDataSet(ds1,aProps.Execute('select * from people',[]));
See sample "{\f1\fs20 17 - TClientDataset use}" to find out more about using such {\f1\fs20 TDataSet}, including some speed information. You need to have run the {\f1\fs20 TestSQL3.dpr} set of regression tests before, to have the expected {\i SQlite3} data file.
:133  TQuery emulation class
The {\f1\fs20 SynDB.pas} unit offers a {\f1\fs20 @**TQuery@}-like class. This class emulates regular {\f1\fs20 TQuery} classes, without inheriting from {\f1\fs20 DB.pas} nor its slow {\f1\fs20 TDataSet}.
It mimics basic {\f1\fs20 TQuery} VCL methods, with the following benefits:
- Does not inherit from {\f1\fs20 TDataset}, but has its own light implementation over {\f1\fs20 SynDB.pas} {\f1\fs20 ISQLDBStatement} result sets, so is usually much faster;
- Will be also faster for field and parameters access by name - or even index;
- Is Unicode-ready, even with older pre-Unicode version of Delphi, able to return the data as {\f1\fs20 WideString}, independently from the current system charset;
- You can still create a {\f1\fs20 TDataSet} from {\f1\fs20 SynDB}'s {\f1\fs20 TQuery}, via the {\f1\fs20 ToDataSet()} function defined in {\f1\fs20 SynDBVCL.pas}.
Of course, since it is not a {\f1\fs20 TDataSet} component, you can not use it directly as a regular replacement for your RAD code.\line But if your application is data-centric and tried to encapsulate its business logic with some classes - i.e. if it tried to properly implement @*OOP@, not RAD - you can still replace directly your existing code with the {\f1\fs20 TQuery} emulator:
!  Q := TQuery.Create(aSQLDBConnection);
!  try
!    Q.SQL.Clear; // optional
!    Q.SQL.Add('select * from DOMAIN.TABLE');
!    Q.SQL.Add('  WHERE ID_DETAIL=:detail;');
!    Q.ParamByName('DETAIL').AsString := '123420020100000430015';
!    Q.Open;
!    Q.First;    // optional
!    while not Q.Eof do begin
!      assert(Q.FieldByName('id_detail').AsString='123420020100000430015');
!      Q.Next;
!    end;
!    Q.Close;    // optional
!  finally
!    Q.Free;
!  end;
You should better use {\f1\fs20 TSQLDBStatement} instead of this wrapper, but having such code-compatible {\f1\fs20 TQuery} replacement could make easier some existing code upgrade, especially for @66@.\line For instance, it will help to avoid deploying the deprecated BDE, generate (much) smaller executable, access any database without paying a big fee, avoid rewriting a lot of existing code lines of a big legacy application, or let your old application communicate with the database over plain HTTP, without the need to install any RDBMS client - see @131@.
:  Storing connection properties as JSON
You can use a {\f1\fs20 TSynConnectionDefinition} storage to persist the connection properties as a @*JSON@ content, in memory or file.
Typical stored content could be:
$ {
$   "Kind": "TSQLDBSQLite3ConnectionProperties",
$   "ServerName": "database.db3",
$   "DatabaseName": "",
$   "UserID": "",
$   "Password": "PtvlPA=="
$ }
The {\f1\fs20 "Kind"} parameter will be used to store the actual {\f1\fs20 TSQLDBConnectionProperties} class name. So switching from one database to another could be easily done at runtime, by modifying a setting stored e.g. in a JSON text file, without the need to recompile the application. Note that the {\f1\fs20 SynDB*} units implementing the class should be compiled within the executable, e.g. {\f1\fs20 SynDBSQLite3.pas} for {\f1\fs20 TSQLDBSQLite3ConnectionProperties}, or {\f1\fs20 SynDBZeos.pas} for {\f1\fs20 TSQLDBZeosConnectionProperties}.
To create a new {\f1\fs20 TSQLDBConnectionProperties} instance from a local JSON file, you could simply write:
!var Props: TSQLDBConnectionProperties;
! ...
!  Props := TSQLDBConnectionProperties.CreateFromFile('localDBsettings.json');
The password will be encrypted and encoded as {\i Base64} in the file, for safety. You could use {\f1\fs20 TSynConnectionDefinition}'s {\f1\fs20 Password} and {\f1\fs20 PasswordPlain} properties to compute the value to be written on disk.
Since {\f1\fs20 TSynConnectionDefinition} is a {\f1\fs20 TSynPersistent} class, you can nest it into a {\f1\fs20 TSynAutoCreateFields} instance containing all settings of your application.\line Then {\f1\fs20 mORMot.pas}' {\f1\fs20 ObjectToJSON}/{\f1\fs20 ObjectToJSONFile} and {\f1\fs20 JSONToObject}/{\f1\fs20 JSONFileToObject} functions could be used for persistence as a file or in a database, of those global settings.
See also {\f1\fs20 TSQLRest.CreateFrom()} for a similar feature at ORM/REST level, and {\f1\fs20 function TSQLRestCreateFrom( aDefinition: TSynConnectionDefinition  )} as defined in {\f1\fs20 mORMotDB.pas} which is able to create a regular local ORM if {\f1\fs20 aDefinition.Kind} is a {\f1\fs20 TSQLRest} class name, but also an ORM with external DB storage - see @146@ - if {\f1\fs20 aDefinition.Kind} is a {\f1\fs20 TSQLDBConnectionProperties} class name.
\page
: SynDB clients
From the {\f1\fs20 SynDB.pas} logical point of view, here is how databases can be accessed:
\graph SynDB1stLevel SynDB First Level Providers
\SynDB\ZDBC
\SynDB\ODBC
\SynDB\OleDB
\SynDB\Oracle
\SynDB\SQLite3
\SynDB\DB.pas¤TDataset
\DB.pas¤TDataset\NexusDB
\DB.pas¤TDataset\BDE
\DB.pas¤TDataset\DBExpress
\DB.pas¤TDataset\FireDAC¤AnyDAC
\DB.pas¤TDataset\UniDAC
\
Of course, the physical implementation is more complicated, as was stated in @%%SynDBLayers@.
We will now detail how these available database connections are interfaced as {\f1\fs20 SynDB.pas} classes.
:118  OleDB or ODBC to rule them all
{\i @**OleDB@} (Object Linking and Embedding, Database, sometimes written as OLE DB or OLE-DB) is an API designed by Microsoft for accessing data from a variety of sources in a uniform manner.
\graph SynDBOleDB SynDB and OleDB
\SynDB\OleDB
\OleDB\Oracle
\OleDB\MS SQL
\OleDB\Jet/Access
\OleDB\MySQL
\OleDB\PostgreSQL
\OleDB\Interbase
\OleDB\Firebird
\OleDB\Sybase
\
;It was designed as a higher-level replacement for, and successor to, ODBC, extending its feature set to support a wider variety of non-relational databases, such as object databases and spreadsheets that do not necessarily implement SQL.
Of course, you have got the {\i Microsoft SQL Native Client} to access the @**MS SQL@ Server 2005/2008/2012, but {\i @*Oracle@} provides a native OleDB provider (even if we found out that this Oracle provider, including the Microsoft's version, have problems with BLOBs). Do not forget about the {\i Advantage Sybase OleDB} driver and such...
If you plan to connect to a MS SQL Server, we highly recommend using the {\f1\fs20 TOleDBMSSQL2012ConnectionProperties} class, corresponding to {\f1\fs20 SQLNCLI11}, part of the {\i Microsoft® SQL Server® 2012 Native Client}, it is able to connect to any revision of MS SQL Server (event MS SQL Server 2008), and was found to be the more stable. You can get it from @http://www.microsoft.com/en-us/download/details.aspx?id=29065 by downloading the {\f1\fs20 sqlncli.msi} corresponding to your Operating System. Most of the time, you should download the {\f1\fs20 X64 Package} of {\f1\fs20 sqlncli.msi}, which will also install the 32-bit version of SQL {\i Server Native Client}, so will work for a 32-bit Delphi executable - the {\f1\fs20 X86 Package} is for a 32-bit Windows system only.
{\i @**ODBC@} ({\i Open DataBase Connectivity}) is a standard C programming language middle-ware API for accessing database management systems (DBMS). {\i ODBC} was originally developed by Microsoft during the early 1990s, then was deprecated in favor to {\i OleDB}. More recently, Microsoft is officially deprecating {\i OleDB}, and urge all developers to switch to the open and cross-platform {\i ODBC} API for native connection. Back & worse strategy from Micro$oft... one more time!\line @http://blogs.msdn.com/b/sqlnativeclient/archive/2011/08/29/microsoft-is-aligning-with-odbc-for-native-relational-data-access.aspx
\graph SynDBODBC SynDB and ODBC
\SynDB\ODBC
\ODBC\Oracle
\ODBC\MS SQL
\ODBC\MySQL
\ODBC\DB2
\ODBC\Informix
\ODBC\PostgreSQL
\ODBC\Interbase
\ODBC\Firebird
\ODBC\Sybase
\ODBC\Jet/Access
\ODBC\Advantage
\
By using our own {\i OleDB} and {\i ODBC} implementations, we will for instance be able to convert directly the {\i OleDB} or {\i ODBC} binary rows to @*JSON@, with no temporary conversion into the {\i Delphi} high-level types (like temporary {\f1\fs20 string} or {\f1\fs20 variant} allocations). The resulting performance is much higher than using standard {\f1\fs20 @*TDataSet@} or other components, since we will bypass most of the layers introduced by BDE/dbExpress/FireDAC/AnyDAC component sets.
Most {\i OleDB / ODBC} providers are free (even maintained by the database owner), others will need a paid license.
It is worth saying that, when used in a {\i mORMot} Client-Server architecture, object persistence using an {\i OleDB} or {\i ODBC} remote access expects only the database instance to be reachable on the Server side. Clients could communicate via standard HTTP, so won't need any specific port forwarding or other IT configuration to work as expected.
:94  ZEOS via direct ZDBC
:   The mORMot's best friend
{\i ZeosLib}, aka {\i @**Zeos@}, is a Open Source library which provides native access to many database systems, developed for {\i Delphi}, {\i Kylix} and {\i @*Lazarus@} / {\i @*FreePascal@}.\line It is fully object-oriented and with a totally modular design. It connects to the databases by wrapping their native client libraries, and makes them accessible via its abstract layer, named {\f1\fs20 @**ZDBC@}. Originally, ZDBC was a port of JDBC 2.0 (Java Database Connectivity API) to Object Pascal. Since that time the API was slightly extended but the main ideas remain unchanged, so official JDBC 2.0 specification is the main entry point to the ZDBC API.
The latest 7.x branch was deeply re-factored, and new methods and performance optimization were introduced. In fact, we worked hand by hand with Michael (the main contributor of {\i ZeosLib}) to ensure that the maximum performance is achieved. The result is an impressive synergy of {\i mORMot} and {\i ZeosLib}, for both reading or writing data.
Since revision 1.18 of the framework, we included direct integration of {\i ZeosLib} into {\i mORMot} persistence layer, with direct access to the {\i ZDBC} layer. That is, our {\f1\fs20 SynDBZeos} unit does not reference {\f1\fs20 DB.pas}, but access directly to the {\i ZDBC} interfaces.
\graph SynDBZeos SynDB and Zeos / ZDBC
\SynDB\ZDBC
\ZDBC\Oracle
\ZDBC\MS SQL
\ZDBC\MySQL
\ZDBC\PostgreSQL
\ZDBC\Interbase
\ZDBC\Firebird
\ZDBC\Sybase
\ZDBC\SQLite3
\
Such direct access, by-passing the VCL {\f1\fs20 DB.pas} layer and its {\f1\fs20 TDataSet} bottleneck, is very close to our {\f1\fs20 SynDB.pas} design. As such, {\i ZeosLib} is a first class citizen library for {\i mORMot}. The {\f1\fs20 SynDBZeos} unit is intended to be a privileged access point to external SQL databases.
:   Recommended version
We recommend that you download the 7.2 branch of {\i Zeos}/ZDBC, which is the current {\i trunk}, at the time of this writing.
A deep code refactoring has been made by the {\i Zeos}/ZDBC authors (thanks a lot Michael, aka {\i EgonHugeist}!), even taking care of {\i mORMot} expectations, to provide the best performance and integration, e.g. for @*UTF-8@ content processing.\line In comparison with the previous 7.1 release, speed increase can be of more than 10 times, depending on the database back-end and use case!
When writing data (i.e. {\i Add/Update/Delete} operations), {\i @*Array bind@ing} suport has been added to the {\i Zeos}/ZDBC 7.2 branch, and our {\f1\fs20 SynDBZeos} unit will use it if available, detecting if {\f1\fs20 IZDatabaseInfo.SupportsArrayBindings} property is {\f1\fs20 true} - which will be the case for {\i @*Oracle@} and {\i @FireBird@} providers by now. Our ORM benefits from it, when processing in BATCH mode, even letting ZDBC creates the optimized SQL - see @99@.\line Performance at reading is very high, much higher than any other {\f1\fs20 DB.pas} based library, in case of single record retrieval. For instance, {\f1\fs20 TSQLDBZEOSStatement.ColumnsToJSON()} will avoid most temporary memory allocation, and is able to create the JSON directly from the low-level ZDBC binary buffers.
If you need to stick to a version prior to 7.2, and want to work as expected with a {\i @*SQlite3@} back-end (but you should'nt have any reason to do so, since {\i Zeos} will be slower compared to {\i SynDBSQlite3}), you need to apply some patches for Zeos < 7.2, in methods {\f1\fs20 TZSQLiteCAPIPreparedStatement. ExecuteQueryPrepared()} and {\f1\fs20 TZSQLiteResultSet. FreeHandle}, as stated as comment at the beginning of {\f1\fs20 SynDBZeos.pas}.
:   Connection samples
If you want e.g. to connect to @*MySQL@ via Zeos/ZDBC, follow those steps:
- Download "{\i Windows (x86, 32-bit), ZIP Archive}" from @http://dev.mysql.com/downloads/connector/c - then extract the archive: only {\f1\fs20 libmysql.dll} is needed and should be placed either in the executable folder, either in the system PATH;
- Connect as usual e.g. via
! fConnection := TSQLDBZEOSConnectionProperties.Create(
!   'zdbc:mysql://192.168.2.60:3306/world?username=root;password=dev', '', '', '');
- Or using the {\f1\fs20 URI()} method:
! fConnection := TSQLDBZEOSConnectionProperties.Create(
!   TSQLDBZEOSConnectionProperties.URI(dMySQL,'192.168.2.60:3306'),'root','dev');
For {\i @*PostgreSQL@}, the {\i Zeos} driver needs only {\f1\fs20 libpq.dll} and {\f1\fs20 libintl.dll} e.g. from @http://www.enterprisedb.com/products-services-training/pgbindownload
! PropsPostgreSQL := TSQLDBZEOSConnectionProperties.Create(
!   TSQLDBZEOSConnectionProperties.URI(dPostgreSQL,'localhost:5432'),
!   'dbname','username','password');
You may therefore use the {\f1\fs20 TSQLDBZEOSConnectionProperties.URI()} method to compute the expected ZDBC connection string:
! PropsOracle := TSQLDBZEOSConnectionProperties.Create(
!   TSQLDBZEOSConnectionProperties.URI(dOracle,'','oci64\oci.dll'),
!   'tnsname','user','pass');
! PropsFirebirdEmbedded := TSQLDBZEOSConnectionProperties.Create(
!   TSQLDBZEOSConnectionProperties.URI(dFirebird,'','Firebird\fbembed.dll')
!   'databasefilename','',');
! PropsFirebirdRemote := TSQLDBZEOSConnectionProperties.Create(
!   TSQLDBZEOSConnectionProperties.URI(dFirebird,'192.168.1.10:3055',
!     'c:\Firebird_2_5\bin\fbclient.dll',false),
!  '3camadas', 'sysdba', 'masterkey');
See {\f1\fs20 TSQLDBZEOSConnectionProperties} documentation for further information about the expected syntax, and available abilities of this great open source library.
:117  Oracle via OCI
For our framework, and in completion to {\f1\fs20 SynDBZeos} or our {\f1\fs20 SynOleDB / SynDBODBC} units, the {\f1\fs20 SynDBOracle} unit has been implemented. It allows {\i direct access} to any remote @**Oracle@ server, using the {\i Oracle Call Interface}.
{\i Oracle Call Interface} (OCI) is the most comprehensive, high performance, native unmanaged interface to the Oracle Database that exposes the full power of the Oracle Database. A direct interface to the {\f1\fs20 oci.dll} library was written, using our DB abstraction classes introduced in {\f1\fs20 @*SynDB@.pas}.
We tried to implement all best-practice patterns detailed in the official {\i Building High Performance Drivers for Oracle} reference document.
Resulting speed is quite impressive: for all requests, {\f1\fs20 SynDBOracle} is 3 to 5 times faster than a {\f1\fs20 SynOleDB} connection using the native {\i OleDB Provider} supplied by Oracle. A similar (even worse) speed penalty has been observed in comparison with the official ODBC driver from Oracle, via a {\f1\fs20 SynDBODBC}-based connection. For more detailed numbers, see @59@.
:   Optimized client library
It is worth saying that, when used in a {\i mORMot} Client-Server architecture, object persistence using an {\i Oracle} database expects only the Oracle instance to be reachable on the Server side, just like with {\i OleDB} or {\i ODBC}.
Here are the main features of this {\f1\fs20 SynDBOracle} unit:
- {\i Direct access} to the {\i Oracle Call Interface} (OCI) client, with no BDE, Midas, DBExpress, nor {\i OleDB / ODBC} provider necessary;
- Dedicated to work with {\i any version} of the Oracle OCI interface, starting from revision 8;
- {\i Optimized for the latest features} of Oracle 11g/12c (e.g. using native {\f1\fs20 Int64} for retrieving NUMBER fields with no decimal);
- Able to work with the {\i Oracle Instant Client} for {\i No Setup} applications (installation via file/folder copy);
- {\i Natively Unicode} (uses internal @*UTF-8@ encoding), for all version of {\i Delphi}, with special handling of each database char-set;
- Tried to achieve {\i best performance available} from every version of the Oracle client;
- Designed to work under {\i any version of Windows}, either in 32 or @*64-bit@ architecture (but the OCI library must be installed in the same version than the compiled {\i Delphi} application, i.e. only 32-bit for this current version);
- {\i @*Late-binding@} access to column names, using a new dedicated {\f1\fs20 Variant} type (similar to Ole Automation runtime properties);
- Connections are {\i multi-thread ready} with low memory and CPU resource overhead;
- Can use connection strings like {\f1\fs20 '//host[:port]/[service_name]'}, avoiding use of the {\f1\fs20 TNSNAME.ORA} file;
- Use {\i Rows Array} and {\i BLOB fetching}, for best performance (ZEOS/ZDBC did not handle this, for instance);
- Handle {\i Prepared Statements} - on both client and server side, if available - server side caching lead to up a 3 times speed boost, from our experiment;
- Implements {\i @*Array Bind@ing} for very fast bulk modifications - insert, update or deletion of a lot of rows at once;
- Implements binding of a {\f1\fs20 TInt64DynArray} or {\f1\fs20 TRawUTF8DynArray} as parameter, e.g. within a {\f1\fs20 SELECT .. IN} where clause;
- {\i Cursor support}, which is pretty common when working with stored procedures and legacy code.
Of course, this unit is perfectly integrated with the @27@ process. For instance, it features native {\i export to @*JSON@} methods, which will be the main entry point for our @*ORM@ framework. And {\i Array binding} is handled directly during @*BATCH@ sequences - see @28@.
:179   Direct connection without Client installation
You can use the latest version of the {\i @**Oracle Instant Client@} (@**OIC@) provided by Oracle - see @http://www.oracle.com/technetwork/database/features/instant-client - which allows to run client applications without installing the standard (huge) Oracle client or having an {\f1\fs20 ORACLE_HOME}.
\graph SynDBOCIdirect Oracle Connectivity with SynDBOracle
\RAD Application\DBExpress¤or BDE
\DBExpress¤or BDE\ installed¤Oracle Client
\ installed¤Oracle Client\ Oracle Server\TCP/IP
\mORMot Application\installed¤Oracle Client
\installed¤Oracle Client\Oracle Server \TCP/IP
\mORMot Application¤with OIC dlls\Oracle Server\TCP/IP
\
Just deliver the few {\f1\fs20 dll} files in the same directory than the application (probably a {\i mORMot} server), and it will work at amazing speed, with all features of Oracle (other stand-alone direct Oracle access library rely on deprecated Oracle 8 protocol).
:   Oracle Wallet support
Password credentials for connecting to databases can now be stored in a client-side {\i @**Oracle Wallet@}, a secure software container used to store authentication and signing credentials.
This wallet usage can simplify large-scale deployments that rely on password credentials for connecting to databases. When this feature is configured, application code, batch jobs, and scripts no longer need embedded user names and passwords. Risk is reduced because such passwords are no longer exposed in the clear, and password management policies are more easily enforced without changing application code whenever user names or passwords change.
In order to use this feature, set {\f1\fs20 TSQLDBOracleConnectionProperties.UseWallet} to {\f1\fs20 true} before connecting to the database.
Wallet configuration is performed on the computer where server is running. You must perform a full Oracle client setup: @*OIC@ - see @179@ - does not give access to wallet authentication.
Steps to create a Wallet:
1) Create a folder for you wallet:
$ > mkdir c:\OraWallets
2) Create a wallet on the client by using the following syntax at the command line:
$ > mkstore -wrl c:\OraWallets -create
Oracle will ask you for the main wallet password - remember it!
3) Create database connection credentials in the wallet by using the following syntax at the command line:
$ mkstore -wrl c:\OraWallets -createCredential TNS_alias_name_from_tnsnames_ora username password
where {\f1\fs20 password} is the password of database user. Oracle will ask you the wallet password - use the main password from previous step.
4) In the client {\f1\fs20 sqlnet.ora} file, add the {\f1\fs20 WALLET_LOCATION} parameter and set it to the directory location of the wallet and set {\f1\fs20 SQLNET.WALLET_OVERRIDE} parameter to {\f1\fs20 TRUE}:
$SQLNET.WALLET_OVERRIDE = TRUE
$WALLET_LOCATION =
$  (SOURCE =
$    (METHOD = FILE)
$    (METHOD_DATA =
$  (DIRECTORY = c:\OraWallets)
$  )
$)
You can not drop a database while it has a wallet. You need to delete wallet credentials via the next command:
$mkstore -wrl wallet_location -deleteCredential db_alias
Oracle will ask to enter the wallet password - use the same password which you used during wallet creation.
Note that there is also an {\i Oracle Wallet Manager} tool available with your database distribution, if you prefer to use a GUI tool for database administration. \line See  @https://docs.oracle.com/cd/B28359_01/network.111/b28530/asowalet.htm
:127  SQLite3
For our ORM framework, we implemented an efficient {\i @*SQLite3@} wrapper, joining the {\i SQLite3} engine either statically (i.e. within the main {\f1\fs20 exe}) or from external {\f1\fs20 sqlite3.dll}.
It was an easy task to let the {\f1\fs20 SynSQLite3.pas} unit be called from our {\f1\fs20 @*SynDB@.pas} database abstract classes. Adding such another Database is just a very thin layer, implemented in the {\f1\fs20 SynDBSQLite3.pas} unit.
If you want to link the {\i SQLite3} engine to your project executable, ensure you defined the {\f1\fs20 SynSQLite3Static.pas} unit in your {\f1\fs20 uses} clause. Otherwise, define a {\f1\fs20 TSQLite3LibraryDynamic} instance to load an external {\f1\fs20 sqlite3.dll} library:
! FreeAndNil(sqlite3); // release any previous instance (e.g. static)
! sqlite3 := TSQLite3LibraryDynamic.Create;
To create a {\i connection property} to an existing {\i SQLite3} database file, call the {\f1\fs20 TSQLDBSQLite3ConnectionProperties. Create} constructor, with the actual {\i SQLite3} database file as {\f1\fs20 ServerName} parameter, and (optionally the proprietary encryption password in {\f1\fs20 Password} - available since rev. 1.16); others ({\f1\fs20 DataBaseName, UserID}) are just ignored.
These classes will implement an internal statement cache, just as the one used for {\f1\fs20 @*TSQLRestServerDB@}. In practice, using the cache can make process up to two times faster (when processing small requests).
When used within the {\i mORMot} ORM, you have therefore two ways of accessing the SQLite3 engine:
- Either directly from the ORM core;
- Either virtually, as external tables.
\graph SynDBSqlite3 SynDB, mORMot and SQLite3
\mORMot ORM\SQLite3
\mORMot ORM\SynDB
\SynDB\SQLite3
\
If your {\i mORMot}-based application purpose is to only use one centralized {\i SQLite3} database, it does not make sense to use {\f1\fs20 SynDBSQLite3} external tables. But if you want, in the future, to be able to connect to any external database, or to split your data in several database files, using those external {\i SQLite3} tables do make sense. Of course, the {\i SQlite3} engine library itself will be shared with both internal and external process.
:119  DB.pas libraries
Since revision 1.18 of the framework, a new {\f1\fs20 SynDBDataset.pas} unit has been introduced, able to interface any {\f1\fs20 DB.pas} based library to our {\f1\fs20 SynDB.pas} classes, using {\f1\fs20 TDataset} to retrieve the results. Due to the {\f1\fs20 TDataset} design, performance is somewhat degraded in respect to direct {\f1\fs20 SynDB.pas} connection (e.g. results for {\i SQLite3} or {\i @*Oracle@}), but it also opens the potential database access.
Some dedicated providers have been published in the {\f1\fs20 SynDBDataset} sub-folder of the {\i mORMot} source code repository. Up to now, {\i @*FireDAC@} (formerly {\i @*AnyDAC@}), {\i @*UniDAC@} and {\i @*BDE@} libraries are interfaced, and a direct connection to the {\i @*NexusDB@} engine is available.
Since there are a lot of potential combinations here - see @%%SynDBLayers@ - feedback is welcome. Due to our Agile process, we will first stick to the providers we need and use. It is up to {\i mORMot} users to ask for additional features, and provide wrappers, if possible, or at least testing abilities. Of course, {\i DBExpress} will benefit to be integrated, even if {\i Embarcadero} just acquired {\i AnyDAC} and revamped/renamed it as {\i FireDAC} - to make it the new official platform.
:   NexusDB access
{\i @**NexusDB@} is a "royalty-free, SQL:2003 core compliant, Client/Server and Embedded database system, with features that rival other heavily licensed products" (vendor's terms).
\graph SynDBNexusDB SynDB and NexusDB
\SynDB\DB
=DB=DB.pas¤TDataset
\DB\NexusDB
\
We used and tested the free embedded edition, which is a perfect match for a Client-Server ORM framework like {\i mORMot} - see @http://www.nexusdb.com/support/index.php?q=FreeEmbedded
:   FireDAC / AnyDAC library
{\i @**FireDAC@} is an unique set of {\i Universal Data Access} Components for developing cross platform database applications on {\i Delphi}. This was in fact a third-party component set, bought by {\i Embarcadero} to {\i DA-SOFT Technologies} (formerly known as @**AnyDAC@), and included with several editions of {\i Delphi} XE3 and up. This is the new official platform for high-speed database development in {\i Delphi}, in favor to the now deprecated {\i DBExpress}.
\graph SynDBFireDAC SynDB and FireDAC / AnyDAC
\SynDB\DB
=DB=DB.pas¤TDataset
\ODBC\Oracle
\ODBC\MS SQL
\ODBC\MySQL
\ODBC\DB2
\ODBC\PostgreSQL
\ODBC\Interbase
\ODBC\Firebird
\ODBC\Sybase
\ODBC\Jet/Access
\ODBC\Advantage
\DB\AnyDAC
\AnyDAC\Oracle
\AnyDAC\MS SQL
\AnyDAC\MySQL
\AnyDAC\PostgreSQL
\AnyDAC\DB2
\AnyDAC\Interbase
\AnyDAC\Firebird
\AnyDAC\ODBC
\AnyDAC\SQLite3
\AnyDAC\Sybase
=AnyDAC=FireDAC¤AnyDAC
\
Our integration within {\f1\fs20 SynDB.pas} units and the {\i mORMot} persistence layer has been tuned. For instance, you can have direct access to high-speed FireDAC {\i Array DML} feature, via the ORM @*batch@ process, via so-called @*array bind@ing - see @78@.
:   UniDAC library
{\i Universal Data Access Components} ({\i @**UniDAC@}) is a cross-platform library of components that provides direct access to multiple databases from {\i Delphi}. See @http://www.devart.com/unidac
\graph SynDBUniDAC SynDB and UniDAC
\SynDB\DB
=DB=DB.pas¤TDataset
\ODBC\Oracle
\ODBC\MS SQL
\ODBC\MySQL
\ODBC\DB2
\ODBC\PostgreSQL
\ODBC\Interbase
\ODBC\Firebird
\ODBC\Sybase
\ODBC\Jet/Access
\ODBC\Advantage
\DB\UniDAC
\UniDAC\Oracle
\UniDAC\ODBC
\UniDAC\MySQL
\UniDAC\OleDB
\OleDB\MS SQL
\UniDAC\DB2
\UniDAC\PostgreSQL
\UniDAC\Interbase
\UniDAC\Firebird
\UniDAC\NexusDB
\UniDAC\SQLite3
\NexusDB=Oracle=SQLite3
\
For instance, to access to a @*MySQL@ remote database, you should be able to connect using:
!PropsMySQL := TSQLDBUniDACConnectionProperties.Create(
!  TSQLDBUniDACConnectionProperties.URI(dMySQL,'192.168.2.60:3306'),
!  'world', 'root', 'dev');
This library gives pretty stable results, but lack of the array binding feature, in comparison to {\i FireDAC}.
:   BDE engine
{\i Borland Database Engine} (@**BDE@) is the Windows-based core database engine and connectivity software shipped with earlier versions of {\i Delphi}. Even if it is deprecated, and replaced by DBExpress since 2000, it is a working solution, easy to interface as a {\f1\fs20 SynDB.pas} provider.
\graph SynDBBDE SynDB and BDE
\SynDB\DB
=DB=DB.pas¤TDataset
\ODBC\Oracle
\ODBC\MS SQL
\ODBC\MySQL
\ODBC\DB2
\ODBC\PostgreSQL
\ODBC\Interbase
\ODBC\Firebird
\ODBC\Sybase
\ODBC\Jet/Access
\ODBC\Advantage
\DB\BDE
\BDE\Paradox
\BDE\Oracle
\BDE\DB2
\BDE\Interbase
\BDE\Informix
\BDE\ODBC
\Oracle=Paradox=Informix
\
Please do not use the BDE on any new project!\line You should better switch to another access layer.
\page
:131  Remote access via HTTP
The {\f1\fs20 SynDBRemote.pas} unit allows you to create database applications that perform SQL operations on a remote HTTP server, instead of a database server. You can create connections just like any other {\f1\fs20 SynDB.pas} database, but the transmission will take place over HTTP. As a result, no database client is to be deployed on the end user application: it will just use HTTP requests, even over Internet. You can use all the features of {\f1\fs20 SynDB.pas} classes, with the ease of one optimized HTTP connection.
\graph SynDBRemoteOverview SynDB Remote access Overview
node [shape=box];
subgraph cluster_0 {
"Client 1";
label="Office 1";
}
subgraph cluster_1 {
"Client 2";
label="Remote A";
}
subgraph cluster_3 {
"Client 3";
label="Office 2";
}
subgraph cluster_4 {
"Client 4";
label="Remote B";
}
subgraph cluster_2 {
\Client 1\DB\direct
\Client 3\DB
\SynDBRemote¤Server\DB\local¤network
\Client 2\SynDBRemote¤Server\HTTP¤HTTPS¤Internet
\Client 4\SynDBRemote¤Server
label="Server";
}
\
This feature is {\i not} part of our @*REST@ful @*ORM@, so does not use the {\f1\fs20 mORMot.pas} unit, but its own optimized protocol, using enhanced security (transmission encryption with user authentication and optional HTTPS) and automatic data compression. Only the HTTP client and server classes, from the {\f1\fs20 SynCrtSock.pas} unit, are used.
Since your application can use both {\f1\fs20 @*TDataSet@} - see @134@ - and emulated {\f1\fs20 @*TQuery@} - see @133@, this new mean of transmission may make it easy to convert existing Delphi client-server applications into @7@ with minimal changes in source code. Then, for your new code, you may switch to a @*SOA@ / @*ORM@ design, using {\i mORMot}'s @*REST@ful abilities - see @6@.
The transmission protocol uses an optimized binary format, which is compressed, encrypted and digitally signed on both ends, and the remote user authentication will be performed via a challenge validation scheme. You can also publish your server over HTTPS, if needed, in {\f1\fs20 http.sys} kernel mode.
:   Server and Client classes
To publish your {\f1\fs20 SynDB.pas} connection, you just need to initialize one of the {\f1\fs20 TSQLDBServer*} classes defined in {\f1\fs20 SynDBRemote.pas}:
\graph HierTSQLDBServerSockets SynDB Remote access Server classes hierarchy
\TSQLDBServerHttpApi\TSQLDBServerAbstract
\TSQLDBServerSockets\TSQLDBServerAbstract
\
You can define either a HTTP server based on the socket API - {\f1\fs20 TSQLDBServerSockets} - or the more stable and fast {\f1\fs20 TSQLDBServerHttpApi} class (under {\i Windows} only), which uses the {\f1\fs20 http.sys} kernel mode HTTP server available since Windows XP - see @88@.
For the client side, you could use one of the following classes also defined in {\f1\fs20 SynDBRemote.pas}:
\graph HierTSQLDBWinINetConnectionProperties SynDB Remote access Client classes hierarchy
\TSQLDBSocketConnectionProperties\TSQLDBHTTPConnectionPropertiesAbstract
\TSQLDBHttpRequestConnectionProperties\TSQLDBHTTPConnectionPropertiesAbstract
\TSQLDBWinHTTPConnectionProperties\TSQLDBHttpRequestConnectionProperties
\TSQLDBWinINetConnectionProperties\TSQLDBHttpRequestConnectionProperties
\TSQLDBCurlConnectionProperties\TSQLDBHttpRequestConnectionProperties
\
Note that {\f1\fs20 TSQLDBHttpRequestConnectionProperties} is an abstract parent class, so you should not instantiate it directly, but one of its inherited implementations.
As you can see, you may choose between a pure socket API client, others using {\f1\fs20 WinINet} or {\f1\fs20 WinHTTP} (under {\i Windows}), or the {\f1\fs20 libcurl} API (especially on {\i @*Linux@}). The {\f1\fs20 TSQLDBWinHTTPConnectionProperties} class is the more stable over the {\i Internet} on {\i Windows}, even if plain sockets tend to give better numbers on {\f1\fs20 localhost} as stated by our @59@. Please read @135@ for a comparison of the diverse APIs.
:   Publish a SynDB connection over HTTP
To define a HTTP server, you may write:
!uses SynDB, // RDBMS core
!     SynDBSQLite3, SynSQLite3Static, // static SQLite3 engine
!     SynDBRemote; // for HTTP server
! ...
!var Props: TSQLDBConnectionProperties;
!    HttpServer: TSQLDBServerAbstract;
! ...
!  Props := TSQLDBSQLite3ConnectionProperties.Create('data.db3','','','');
!!  HttpServer := TSQLDBServerHttpApi.Create(Props,'syndbremote','8092','user','pass');
The above code will initialize a connection to a local {\f1\fs20 data.db3} {\i SQlite3} database (in the {\f1\fs20 Props} variable), and then publish it using the {\f1\fs20 http.sys} kernel mode HTTP server to the {\f1\fs20 http://1.2.3.4:8092/syndbremote} URI - if the server's IP is {\f1\fs20 1.2.3.4}.
A first user is defined, with '{\f1\fs20 user}' / '{\f1\fs20 pass}' credentials. Note that in our remote access, user management {\i does not} match the RDBMS user rights: you should better have your own set of users at application level, for higher security, and a better integration with your business logic. If creating a new user on a RDBMS could be painful, managing remote user @*authentication@ is pretty easy on the {\f1\fs20 SynDBRemote.pas} side, by using the {\f1\fs20 Protocol.Authenticate} property of the server:
!HttpServer.Protocol.Authenticate.AuthenticateUser('toto','pipo');
!HttpServer.Protocol.Authenticate.AuthenticateUser('toto2','pipo2');
!...
You could also share {\i mORMot}'s REST authentication users @139@, by replacing the default {\f1\fs20 TSynAuthentication} class instance with {\f1\fs20 TSynAuthenticationRest}, as defined in {\f1\fs20 mORMot.pas}. Note using at the same time {\f1\fs20 SynDBRemote} and {\i mORMot}'s ORM/SOA sounds like a weak design, but may have its benefits when dealing with legacy code, and a lot of existing SQL statements.
The URI should be registered to work as expected, just as expected by the {\f1\fs20 http.sys} API - see @109@. You may either run the server once with the system Administrator rights, or call the following method (as we do in {\f1\fs20 TestSQL3Register.dpr}) in your setup application:
!THttpApiServer.AddUrlAuthorize('syndbremote','8092',false,'+');
:   SynDB client access via HTTP
On the client side, you can then write:
!uses SynDB, // RDBMS core
!     SynDBRemote; // for HTTP client
! ...
!var Props: TSQLDBConnectionProperties;
! ...
!!  Props := TSQLDBWinHTTPConnectionProperties.Create('1.2.3.4:8092','syndbremote','user','pass');
As you can see, there is no link to {\f1\fs20 SynDBSQLite3.pas} nor {\f1\fs20 SynSQLite3Static.pas} on the client side. Just the HTTP link is needed. No need to deploy the RDBMS client libraries with your application, nor setup the local network firewall.
We defined here a single user, with 'user' / 'pass' credentials, but you may manage more users on the server side, using the {\f1\fs20 Protocol.Authenticate} property of {\f1\fs20 TSQLDBServerAbstract}.
Then, you execute your favorite SQL using the connection just as usual:
!procedure Test(Props: TSQLDBConnectionProperties);
!var Stmt: ISQLDBRows;
!begin
!  Stmt := Props.Execute('select * from People where YearOfDeath=?',[1519]);
!  while Stmt.Step do begin
!    assert(Stmt.ColumnInt('ID')>0);
!    assert(Stmt.ColumnInt('YearOfDeath')=1519);
!   end;
!end;
Or you may use it with VCL components, using the {\f1\fs20 SynDBVCL.pas} unit:
!  ds1.DataSet.Free; // release previous TDataSet
!  ds1.DataSet := ToDataSet(ds1,Props.Execute('select * from people',[]));
The {\f1\fs20 TSynSQLStatementDataSet} result set will map directly the raw binary data returned by the {\f1\fs20 TSQLDBServer*} class, avoiding any slow data marshalling in your client application, even for huge content. Note that all the whole data is computed and sent by the server: even if you display only the first rows in your {\f1\fs20 TDBGrid}, all the data has been transmitted. In fact, partial retrieval works well on a local network, but is not a good idea over the Internet, due to its much higher {\f1\fs20 ping}. So consider adding some filter fields, or some application-level paging, to reduce the number of rows retrieved from the {\f1\fs20 SynDBRemote} server.
If you defined you own {\f1\fs20 TSynAuthentication} class on the server class (e.g. to use REST users and groups via {\f1\fs20 TSynAuthenticationRest}), you should create you own class, and override the following method:
!procedure TSQLDBWinHTTPConnectionPropertiesRest.SetInternalProperties;
!begin
!  if fProtocol=nil then
!    fProtocol := TSQLDBRemoteConnectionProtocol.Create(
!!      TSynAuthenticationRest.Create(nil,[]));
!  inherited;
!end;
This overriden method will inherit from {\f1\fs20 TSQLDBWinHTTPConnectionProperties} all its behavior, but use the ORM/SOA authentication scheme for validating its users on the server side.
:   Advanced use cases
You may use this remote connection feature e.g. to mutate a stand-alone shared {\i SQLite3} database into a high performance but low maintenance client-server database engine. You may create it as such on the server side:
!var props: TSQLDBSQLite3ConnectionProperties;
!    server: TSQLDBServerHttpApi;
!...
!  props := TSQLDBSQLite3ConnectionProperties.Create('database.db3','','','');
!  props.MainSQLite3DB.Synchronous := smOff;
!  props.MainSQLite3DB.LockingMode := lmExclusive; // tune the performance
!  server := TSQLDBServerHttpApi.Create(props,'syndbremote','8092','user','password');
!...
You could share an existing {\i SQlite3} database instance (e.g. a {\f1\fs20 @*TSQLRestServerDB@} used for our @*REST@ful ORM - see @42@) by creating the properties as such:
!!  props := TSQLDBSQLite3ConnectionProperties.Create(aRestServerDB.DB);
!  server := TSQLDBServerHttpApi.Create(props,'syndbremote','8092','user','password');
If you use the {\f1\fs20 http.sys} kernel-mode server, you could share the same IP port between regular ORM/SOA operations (which may be 80 for a "pure" HTTP server), and remote {\f1\fs20 SynDB} access, if the database name (i.e. here '{\f1\fs20 syndbremote}') does not conflict with a ORM table nor a method service.
Note that you can also customize the transmission protocol by setting your own {\f1\fs20 TSQLDBProxyConnectionProtocol} class on both server and server sides.
:   Integration with SynDBExplorer
Our {\f1\fs20 @*SynDBExplorer@} tool is able to publish in one click any {\f1\fs20 SynDB} connection as a HTTP server, or connect to it via HTTP. It could be very handy, even for debugging purposes.
To serve an existing database, just connect to it as usual. Then click on the "{\f1\fs20 HTTP Server}" button below the table lists (on left side). You can tune the server properties (HTTP port, database name used for URI, user credentials), then click on the "{\f1\fs20 Start}" button.
To connect to this remote connection, run another instance of {\f1\fs20 SynDBExplorer}. Create a new connection, using "{\f1\fs20 Remote HTTP}" as connection type, and set the other options with the values matching the server side, e.g. with the default "{\f1\fs20 localhost:8092}" (replacing {\f1\fs20 localhost} with the server IP for an access over the network) for the server name, "{\f1\fs20 syndbremote}" for the database name, and "{\f1\fs20 synopse}" for both user name and password.
You will be able to access the main server instance remotely, just as if the database was accessed via a regular client.
If the server side database is {\i SQLite3}, you just mutated this local engine into a true client-server database - you may be amazed by the resulting performance.
:   Do not forget the mORMot!
Even if you may be tempted to use such remote access to implement a {\i n-Tier architecture}, you should rather use {\i mORMot}'s Client-Server @*ORM@ instead - see @114@ - which offers much better client-server integration - due to the {\i @*Persistence Ignorance@} pattern of @54@, a better @*OOP@ and @*SOLID@ modeling design - see @47@, and even higher performance than raw SQL operations - see e.g. @28@ or @39@. Our little {\i mORMot} is not an ORM on which we added a data transmission layer: it is a full @*REST@ful system, with a true @*SOA@ design.
But for integrating some legacy SQL code into a new architecture, {\f1\fs20 SynDBRemote.pas} may have its benefits, used in conjunction with {\i mORMot}'s higher level features.
Note that for cross-platform clients, {\i mORMot}'s ORM/SOA patterns are a much better approach: do not put SQL in your mobile application, but use services, so that you will not need to re-validate and re-publish the app to the store after any small fix of your business logic!
\page
:146 SynDB ORM Integration
:  Code-first or database-first
When working with any @13@, you have mainly two possibilities:
- Start from scratch, i.e. write your classes and let the ORM create all the database structure, which will reflect directly the object properties - it is also named "@*code-first@";
- Use an existing database, and then define in your model how your classes map the existing database structure - this is the "@*database-first@" option.
Our {\i mORMot} framework implements both paths, even if, like for other ORMs, code-first sounds like a more straight option.
:145  Code-first ORM
An {\i external} record can be defined as such, as expected by {\i mORMot}'s @*ORM@:
!type
!  TSQLRecordPeopleExt = class(TSQLRecord)
!  private
!    fData: TSQLRawBlob;
!    fFirstName: RawUTF8;
!    fLastName: RawUTF8;
!    fYearOfBirth: integer;
!    fYearOfDeath: word;
!    fLastChange: TModTime;
!    fCreatedAt: TCreateTime;
!  published
!    property FirstName: RawUTF8 index 40 read fFirstName write fFirstName;
!    property LastName: RawUTF8 index 40 read fLastName write fLastName;
!    property Data: TSQLRawBlob read fData write fData;
!    property YearOfBirth: integer read fYearOfBirth write fYearOfBirth;
!    property YearOfDeath: word read fYearOfDeath write fYearOfDeath;
!    property LastChange: TModTime read fLastChange write fLastChange;
!    property CreatedAt: TCreateTime read fCreatedAt write fCreatedAt;
!  end;
As you can see, there is no difference with an {\i internal} ORM class: it inherits from {\f1\fs20 @*TSQLRecord@}, but you may want it to inherit from {\f1\fs20 TSQLRecordMany} to use @58@ for instance.
The only difference is this {\f1\fs20 @**index@ 40} attribute in the definition of {\f1\fs20 FirstName} and {\f1\fs20 LastName} @*published properties@: this will define the length (in {\i UTF-16} {\f1\fs20 WideChar} or {\i UTF-8} bytes) to be used when creating the external field for TEXT column. See above e.g.:
!    property FirstName: RawUTF8
!!      index 40
!      read fFirstName write fFirstName;
In fact, {\i @*SQLite3@} does not care about textual field length, but almost all other database engines expect a maximum length to be specified when defining a {\f1\fs20 VARCHAR} column in a table. If you do not specify any length in your field definition (i.e. if there is no {\f1\fs20 index ???} attribute), the ORM will create a column with an unlimited length (e.g. {\f1\fs20 varchar(max)} for {\i @*MS SQL@ Server}). In this case, code will work, but performance and disk usage may be highly degraded, since access via a CLOB is known to be notably slower. The only exceptions to this performance penalty are {\i SQlite3} and {\i PostgreSQL}, for which the size unlimited {\f1\fs20 TEXT} columns are as fast to process than {\f1\fs20 varchar(#)}.
By default, no check will be performed by the ORM to ensure that the field length is compliant with the column size expectation in the external database. You can use {\f1\fs20 TSQLRecordProperties}'s {\f1\fs20 SetMaxLengthValidatorForTextFields()} or {\f1\fs20 SetMaxLengthFilterForTextFields()} method to create a validation or filter rule to be performed before sending the data to the external database - see @56@.
Here is an extract of the regression test corresponding to external databases:
!var RExt: TSQLRecordPeopleExt;
!  (...)
!fProperties := TSQLDBSQLite3ConnectionProperties.Create(SQLITE_MEMORY_DATABASE_NAME,'','','');
!!VirtualTableExternalRegister(fExternalModel,TSQLRecordPeopleExt,fProperties,'PeopleExternal');
!aExternalClient := TSQLRestClientDB.Create(fExternalModel,nil,'testExternal.db3',TSQLRestServerDB);
!try
!  aExternalClient.Server.StaticVirtualTableDirect := StaticVirtualTableDirect;
!  aExternalClient.Server.CreateMissingTables;
!  Check(aExternalClient.Server.CreateSQLMultiIndex(
!    TSQLRecordPeopleExt,['FirstName','LastName'],false));
!  (...)
!  Start := aExternalClient.ServerTimestamp;
!  (...)
!  aID := aExternalClient.Add(RExt,true);
!  (...)
!  aExternalClient.Retrieve(aID,RExt);
!  (...)
!  aExternalClient.BatchStart(TSQLRecordPeopleExt);
!  aExternalClient.BatchAdd(RExt,true);
!  (...)
!  Check(aExternalClient.BatchSend(BatchID)=HTTP_SUCCESS);
!  Check(aExternalClient.TableHasRows(TSQLRecordPeopleExt));
!  Check(aExternalClient.TableRowCount(TSQLRecordPeopleExt)=n);
!  (...)
!  RExt.FillPrepare(aExternalClient,'FirstName=? and LastName=?',
!    [RInt.FirstName,RInt.LastName]); // query will use index -> fast :)
!  while RExt.FillOne do ...
!  (...)
!  Updated := aExternalClient.ServerTimestamp;
!  (...)
!  aExternalClient.Update(RExt);
!  aExternalClient.UnLock(RExt);
!  (...)
!  aExternalClient.BatchStart(TSQLRecordPeopleExt);
!  aExternalClient.BatchUpdate(RExt);
!  (...)
!  aExternalClient.BatchSend(BatchIDUpdate);
!  (...)
!  aExternalClient.Delete(TSQLRecordPeopleExt,i)
!  (...)
!  aExternalClient.BatchStart(TSQLRecordPeopleExt);
!  aExternalClient.BatchDelete(i);
!  (...)
!  aExternalClient.BatchSend(BatchIDUpdate);
!  (...)
!  for i := 1 to BatchID[high(BatchID)] do begin
!    RExt.fLastChange := 0;
!    RExt.CreatedAt := 0;
!    RExt.YearOfBirth := 0;
!    ok := aExternalClient.Retrieve(i,RExt,false);
!    Check(ok=(i and 127<>0),'deletion');
!    if ok then begin
!      Check(RExt.CreatedAt>=Start);
!      Check(RExt.CreatedAt<=Updated);
!      if i mod 100=0 then begin
!        Check(RExt.YearOfBirth=RExt.YearOfDeath,'Update');
!        Check(RExt.LastChange>=Updated);
!      end else begin
!        Check(RExt.YearOfBirth<>RExt.YearOfDeath,'Update');
!        Check(RExt.LastChange>=Start);
!        Check(RExt.LastChange<=Updated);
!      end;
!    end;
!  end;
!  (...)
As you can see, there is no difference with using the local {\i @*SQLite3@} engine or a remote database engine.\line From the Client point of view, you just call the usual @*REST@ful @*CRUD@ methods, i.e. {\f1\fs20 Add() Retrieve() Update() UnLock() Delete()} - or their faster {\f1\fs20 Batch*()} revision - and you can even handle advanced methods like a {\f1\fs20 FillPrepare} with a complex WHERE clause, or {\f1\fs20 CreateSQLMultiIndex / CreateMissingTables} on the server side.
Even the creation of the table in the remote database (the {\f1\fs20 'CREATE TABLE...'} SQL statement) is performed by the framework when the {\f1\fs20 CreateMissingTables} method is called, with the appropriate column properties according to the database expectations (e.g. a {\f1\fs20 TEXT} for {\i SQLite3} will be a {\f1\fs20 NVARCHAR2} field for {\i @*Oracle@}).
The resulting table layout on the external database will be the following:
\graph TSQLRecordPeopleExtDefaultMapping TSQLRecordPeopleExt Code-First Field/Column Mapping
rankdir=LR;
node [shape=Mrecord];
struct1 [label="<id>ID : TID|<data>Data : TSQLRawBlob|<fn>FirstName : RawUTF8|<ln>LastName : RawUTF8|<yob>YearOfBirth : integer|<yod>YearOfDeath : word"];
struct2 [label="<id>ID : INTEGER|<data>Data : BLOB|<fn>FirstName : NVARCHAR(40)|<ln>LastName : NVARCHAR(40)|<yob>YearOfBirth : INTEGER|<yod>YearOfDeath : INTEGER"];
struct1:id -> struct2:id;
struct1:data -> struct2:data;
struct1:fn -> struct2:fn;
struct1:ln -> struct2:ln;
struct1:yob -> struct2:yob;
struct1:yod -> struct2:yod;
\
The only specific instruction is the global {\f1\fs20 @*VirtualTableExternalRegister@()} function, which has to be run on the server side (it does not make any sense to run it on the client side, since for the client there is no difference between any tables - in short, the client do not care about storage; the server does).
In order to work as expected, {\f1\fs20 VirtualTableExternalRegister()} shall be called {\i before} {\f1\fs20 TSQLRestServer.Create} constructor: when the server initializes, the ORM server must know whenever an {\i internal} or {\i external} database shall be managed. In the above code, {\f1\fs20 TSQLRestClientDB.Create()} will instantiate its own embedded {\f1\fs20 TSQLRestServerDB} instance.
Note that the {\f1\fs20 TSQLRecordExternal.LastChange} field was defined as a {\f1\fs20 TModTime}: in fact, the current date and time will be stored each time the record is updated, i.e. for each {\f1\fs20 aExternalClient.Add} or {\f1\fs20 aExternalClient.Update} calls. This is tested by both {\f1\fs20 RExt.LastChange>=Start} and {\f1\fs20 RExt.LastChange<=Updated} checks in the latest loop. The time used is the "server-time", i.e. the current time and date on the server (not on the client), and, in the case of external databases, the time of the remote server (it will execute e.g. a {\f1\fs20 select getdate()} under @*MS SQL@ to synchronize the date to be inserted for {\f1\fs20 LastChange}). In order to retrieve this server-side time stamp, we use {\f1\fs20 Start := aExternalClient.ServerTimestamp} instead of the local {\f1\fs20 TimeLogNow} function.
A similar feature is tested for the {\f1\fs20 CreatedAt} published field, which was defined as {\f1\fs20 TCreateTime}: it will be set automatically to the current server time at record creation (and not changed on modifications). This is the purpose of the {\f1\fs20 RExt.CreatedAt<=Updated} check in the above code.
:120  Database-first ORM
As we have just seen, the following line initializes the ORM to let {\f1\fs20 TSQLRecordPeopleExt} data be accessed via SQL, over an external database connection {\f1\fs20 fProperties}:
!VirtualTableExternalRegister(fExternalModel,TSQLRecordPeopleExt,fProperties,'PeopleExternal');
We also customized the name of the external table, from its default {\f1\fs20 'PeopleExt'} (computed by trimming {\f1\fs20 TSQLRecord} prefix from {\f1\fs20 TSQLRecordPeopleExt}) into {\f1\fs20 'PeopleExternal'}.
In addition to table name @**mapping@, the ORM is also able to map the {\f1\fs20 TSQLRecord} published properties names to any custom database column name. It is in fact very common that most tables on existing databases to not have very explicit column naming, which may sounds pretty weird when mapped directly as {\f1\fs20 TSQLRecord} property names. Even the @*primary key@s of your existing database won't match the ORM's requirement of naming it as {\f1\fs20 ID}. All this should be setup as expected.
By default, for a {\i code-driven} approach, internal property names will match the external table column names - see @%%TSQLRecordPeopleExtDefaultMapping@
You can customize this default mapping, writing e.g.
!fProperties := TSQLDBSQLite3ConnectionProperties.Create(SQLITE_MEMORY_DATABASE_NAME,'','','');
!VirtualTableExternalRegister(fExternalModel,TSQLRecordPeopleExt,fProperties,'PeopleExternal');
!!fExternalModel.Props[TSQLRecordPeopleExt].ExternalDB.
!!  MapField('ID','Key').
!!  MapField('YearOfDeath','YOD');
! (...) // the remaining code stays the same
As an alternative, you can use the {\f1\fs20 @*VirtualTableExternalMap@} function and its fluent interface:
!fProperties := TSQLDBSQLite3ConnectionProperties.Create(SQLITE_MEMORY_DATABASE_NAME,'','','');
!!VirtualTableExternalMap(fExternalModel,TSQLRecordPeopleExt,fProperties,'PeopleExternal').
!!  MapField('ID','Key').
!!  MapField('YearOfDeath','YOD');
Then you use your {\f1\fs20 TSQLRecordPeopleExt} table as usual from {\i Delphi} code, with {\f1\fs20 ID} and {\f1\fs20 YearOfDeath} fields.
But, under the hood, the mORMot ORM will do the mapping when creating all needed SQL statements:
- The "internal" {\f1\fs20 TSQLRecord} class will be stored within the {\f1\fs20 PeopleExternal} external table;
- The "internal" {\f1\fs20 TSQLRecord.ID} field will be an external "{\f1\fs20 Key: INTEGER}" column;
- The "internal" {\f1\fs20 TSQLRecord.YearOfDeath} field will be an external "{\f1\fs20 YOD: INTEGER}" column;
- Other internal published properties will be mapped by default with the same name to external column.
The resulting mapping will therefore be the following:
\graph TSQLRecordPeopleExtCustomMapping TSQLRecordPeopleExt Database-First Field/Column Mapping
rankdir=LR;
node [shape=Mrecord];
struct1 [label="<id>ID : TID|<data>Data : TSQLRawBlob|<fn>FirstName : RawUTF8|<ln>LastName : RawUTF8|<yob>YearOfBirth : integer|<yod>YearOfDeath : word"];
struct2 [label="<id>Key : INTEGER|<data>Data : BLOB|<fn>FirstName : NVARCHAR(40)|<ln>LastName : NVARCHAR(40)|<yob>YearOfBirth : INTEGER|<yod>YOD : INTEGER"];
struct1:id -> struct2:id;
struct1:data -> struct2:data;
struct1:fn -> struct2:fn;
struct1:ln -> struct2:ln;
struct1:yob -> struct2:yob;
struct1:yod -> struct2:yod;
\
Note that only the {\f1\fs20 ID} and {\f1\fs20 YearOfDeath} column names were customized.
Due to the design of {\i SQLite3} virtual tables, and {\i mORMot} internals in its current state, the database @*primary key@ must be an {\f1\fs20 INTEGER} field to be mapped as expected by the ORM. But you can specify any secondary key, e.g. a {\f1\fs20 TEXT} field, via {\f1\fs20 stored @*AS_UNIQUE@} definition in code.
:106  Sharing the database with legacy code
It is pretty much possible that you will have to maintain and evolve a legacy project, based on an existing database, with a lot of already written SQL statements - see @66@. For instance, you would like to use {\i mORMot} for new features, and/or add mobile or HTML clients - see @86@.\line In this case, the @*ORM@ advanced features - like @38@ or BATCH process, see @28@ - may conflict with the legacy code, for the tables which may have to be shared. Here are some guidelines when working on such a project.
To be exhaustive about this question, we need to consider each ORM @*CRUD@ operation. We may have to divide them in three kinds: read queries, insertions, and modifications of existing data.
About {\b ORM read queries}, i.e. {\f1\fs20 Retrieve()} methods, the ORM cache can be tuned per table, and you will definitively lack of some cache, but remember :
- That you can set a "time out" period for this cache, so that you may still benefit of it in most cases;
- That you have a cache at server level and another at client level, so you can tune it to be less aggressive on the client, for instance;
- That you can tune the ORM cache {\i per ID}, so some items which are not likely to change can still be cached.
About {\b ORM insertions}, i.e. {\f1\fs20 Add()} or {\f1\fs20 BatchAdd()} methods, when using the external engine, if any external process is likely to INSERT new rows, ensure you set the {\f1\fs20 TSQLRestStorageExternal} {\f1\fs20 EngineAddUseSelectMaxID} property to TRUE, so that it will compute the next maximum ID by hand.\line But it still may be an issue, since the external process may do an INSERT {\i during} the ORM insertion.\line So the best is perhaps to NOT use the ORM {\f1\fs20 Add()} or {\f1\fs20 BatchAdd()} methods, but rely on dedicated INSERT SQL statement, e.g. hosted in an interface-based service on the server side.
About {\b ORM modifications}, i.e. {\f1\fs20 Update() Delete() BatchUpdate() BatchDelete()} methods, they sound safe to be used in conjunction with external process modifying the DB, as soon as you use transactions to let the modifications be atomic, and won't conflict any concurrent modifications in the legacy code.
Perhaps the safer pattern, when working with external tables which are to be modified in the background by some legacy code, may be to by-pass those ORM methods, and define server-side {\i interface-based services} - see @63@. Those services may contain manual SQL, instead of using the ORM "magic". But it will depend on your business logic, and you will fail to benefit from the ORM features of the framework.\line Nevertheless, introducing @17@ into your application will be very beneficial: ORM is not mandatory, especially if you are "fluent" in SQL queries, know how to make them as standard as possible, and have a lot of legacy code, perhaps with already tuned SQL statements.
Introducing @*SOA@ is mandatory to interface new kind of clients to your applications, like mobile apps or AJAX modern sites. To be fair, you should not access directly the database any more, as you did with your legacy Delphi application and @*RAD@ DB components.\line All new features, involving new tables to store new data, will still benefit of the {\i mORMot}'s ORM, and could still be hosted in the very same external database, shared by your existing code.\line Then, you will be able to identify {\i seams} - see @66@ - in your legacy code, and move them to your new {\i mORMot} services, then let your application evolve into a newer SOA/MVC architecture, without breaking anything, nor starting from scratch.
:  Auto-mapping of SQL conflictual field names
If your application is likely to be run on several databases, it may be difficult to handle any potential field name conflict, when you switch from one engine to another. The ORM allows you therefore to ensure that no field name will conflict with a SQL keyword of the underlying database.
In code-first mode, you can use the following method to ensure that no such conflict occurs:
!  fExternalModel.Props[TSQLRecordPeopleExt].ExternalDB.MapAutoKeywordFields;
For a database-first database, the following syntax is to be used so that field names will be checked:
!  fExternalModel.Props[TSQLRecordPeopleExt].ExternalDB. // custom field mapping
!    MapField('ID','Key').
!    MapField('YearOfDeath','YOD').
!!    MapAutoKeywordFields;
or, if you want to full fluent interface definition:
!  VirtualTableExternalMap(fExternalModel,TSQLRecordPeopleExt,fProperties,'PeopleExternal').
!    MapField('ID','Key').
!    MapField('YearOfDeath','YOD').
!!    MapAutoKeywordFields
It is a good idea to call the {\f1\fs20 @**MapAutoKeywordFields@} method after any manual field mapping for a database-first database, since even your custom field names may conflict with a SQL keyword.
If any field name is likely to conflict with a SQL keyword, it will be mapped with a trailing '_'. For instance, a {\f1\fs20 'Select'} published property will be mapped into a {\f1\fs20 SELECT_} column in the table.
Even if this option is disabled by default, a warning message will appear in the log proposing to use this {\f1\fs20 MapAutoKeywordFields} method, and will help you to identify such issues.
:30  External database ORM internals
The {\f1\fs20 mORMotDB.pas} unit implements @*Virtual Table@s access for any {\f1\fs20 @*SynDB@.pas}-based external database for the framework.
In fact, this feature will use {\f1\fs20 TSQLRestStorageExternal, TSQLVirtualTableCursorExternal} and {\f1\fs20 TSQLVirtualTableExternal} classes, defined as such:
\graph HierExternalTables External Databases classes hierarchy
\TSQLRecordVirtual\TSQLRecord
\TSQLRecordVirtualTableAutoID\TSQLRecordVirtual
\TSQLVirtualTableCursorExternal\TSQLVirtualTableCursor
\TSQLVirtualTableExternal\TSQLVirtualTable
\
The registration of the class is done by a call to the following new global procedure:
!procedure VirtualTableExternalRegister(aModel: TSQLModel; aClass: TSQLRecordClass;
!  aExternalDB: TSQLDBConnectionProperties; const aExternalTableName: RawUTF8);
This procedure will register on the Server-side an external database for an ORM class:
- It will define the supplied class to behave like a {\f1\fs20 TSQLRecordVirtualTableAutoID} class (i.e. its {\f1\fs20 @*TSQLModelRecordProperties@.Kind} property will be overwritten to {\f1\fs20 rCustomAutoID} in this ORM model);
- It will associate the supplied class with a {\f1\fs20 TSQLVirtualTableExternal} module;
- The {\f1\fs20 TSQLDBConnectionProperties} instance should be shared by all classes, and released globally when the ORM is no longer needed;
- The full table name, as expected by the external database, should be provided here ({\f1\fs20 SQLTableName} will be used internally as table name when called via the associated {\i SQLite3} Virtual Table) - if no table name is specified (''), will use {\f1\fs20 SQLTableName} (e.g. 'Customer' for a class named {\f1\fs20 TSQLCustomer});
- Internal adjustments will be made to convert SQL on the fly from internal ORM representation into the expected external SQL format (e.g. table name or {\f1\fs20 ID} property) - see {\f1\fs20 TSQLRestStorage. AdaptSQLForEngineList} method.
Typical usage may be for instance:
!aProps := TOleDBMSSQLConnectionProperties.Create('.\SQLEXPRESS','AdventureWorks2008R2','','');
!aModel := TSQLModel.Create([TSQLCustomer],'root');
!VirtualTableExternalRegister(aModel,TSQLCustomer,aProps,'Sales.Customer');
!aServer := TSQLRestServerDB.Create(aModel,'application.db'),true)
All the rest of the code will use the "regular" ORM classes, methods and functions, as stated by @3@.
In order to be stored in an external database, the ORM records can inherit from any {\f1\fs20 TSQLRecord} class. Even if this class does not inherit from {\f1\fs20 TSQLRecordVirtualTableAutoID}, it will behave as such, once {\f1\fs20 @*VirtualTableExternalRegister@} function has been called for the given class.
As with any regular {\f1\fs20 TSQLRecord} classes, the ORM core will expect external tables to map an {\f1\fs20 Integer ID} published property, auto-incremented at every record insertion. Since not all databases handle such fields - e.g. {\i @*Oracle@} - auto-increment will be handled via a {\f1\fs20 select max(id) from tablename} statement run at initialization, then computed on the fly via a thread-safe cache of the latest inserted {\i RowID}.
You do not have to know where and how the data persistence is stored. The framework will do all the low-level DB work for you. And thanks to the {\i Virtual Table} feature of {\i SQlite3}, internal and external tables can be mixed within SQL statements. Depending on the implementation needs, classes could be persistent either via the internal {\i SQLite3} engine, or via external databases, just via a call to {\f1\fs20 VirtualTableExternalRegister()} before server initialization.
In fact, {\f1\fs20 TSQLVirtualTableCursorExternal} will convert any query on the external table into a proper optimized SQL query, according to the indexes existing on the external database. {\f1\fs20 TSQLVirtualTableExternal} will also convert individual SQL modification statements (like insert / update / delete) at the {\i SQLite3} level into remote SQL statements to the external database.
Most of the time, all @*REST@ful methods ({\f1\fs20 GET/POST/PUT/DELETE}) will be handled directly by the {\f1\fs20 TSQLRestStorageExternal} class, and won't use the virtual table mechanism. In practice, most access to the external database will be as fast as direct access, but the virtual table will always be ready to interpret any cross-database complex request or statement.
Direct REST access will be processed as following - when adding an object, for instance:
\graph TSQLRecordPeopleExtVirtualRest ORM Access Via REST
\TSQLRestServerDB.Add\TSQLRestServerDB.EngineAdd\internal¤table
\TSQLRestServerDB.EngineAdd\TSQLRequest\INSERT INTO...
\TSQLRequest\SQlite3 engine\internal engine
\SQlite3 engine\SQLite3 file
\TSQLRestServerDB.Add\TSQLRestStorageExternal.EngineAdd\external¤table¤REST
\TSQLRestStorageExternal.EngineAdd\ISQLDBStatement\INSERT INTO...
\ISQLDBStatement\External DB client\ODBC/ZDBC/OleDB...
\External DB client\External DB server
\
\page
Indirect access via virtual tables will be processed as following:
\graph TSQLRecordPeopleExtVirtualVirtual ORM Access Via Virtual Table
\TSQLRestServerDB.Add\TSQLRestServerDB.EngineAdd\internal or¤external table
\TSQLRestServerDB.EngineAdd\TSQLRequest\INSERT INTO...
\TSQLRequest\SQlite3 engine\internal engine
\SQlite3 engine\SQlite3 file\internal¤table
\SQlite3 engine\TSQLVirtualTableExternal.Insert\external¤table
\TSQLVirtualTableExternal.Insert\ISQLDBStatement\INSERT INTO...
\ISQLDBStatement\External DB client\ODBC/ZDBC/OleDB...
\External DB client\External DB server
\
About speed, here is an extract of the test regression log file (see code above, in previous paragraph), which shows the difference between RESTful call and virtual table call, working with more than 11,000 rows of data:
$  - External via REST: 133,666 assertions passed  409.82ms
$  - External via virtual table: 133,666 assertions passed  1.12s
The first run is made with {\f1\fs20 TSQLRestServer.StaticVirtualTableDirect} set to TRUE (which is the default) - i.e. it will call directly {\f1\fs20 TSQLRestStorageExternal} for RESTful commands, and the second will set this property to FALSE - i.e. it will call the {\i SQLite3} engine and let its virtual table mechanism convert it into another SQL calls.
It is worth saying that this test is using an in-memory {\i SQLite3} database (i.e. instantiated via {\f1\fs20 SQLITE_MEMORY_DATABASE_NAME} as pseudo-file name) as its external DB, so what we test here is mostly the ORM overhead, not the external database speed. With real file-based or remote databases (like @*MS SQL@), the overhead of remote connection won't make noticeable the use of Virtual Tables.
In all cases, letting the default {\f1\fs20 StaticVirtualTableDirect=true} will ensure the best possible performance. As stated by @59@, using a virtual or direct call won't affect the CRUD operation speed: it will by-pass the virtual engine whenever possible.
:  Tuning the process
@*Multi-thread@ing abilities of the server, and all available settings, will be detailed @25@.\line By default, all @*ORM@ read operations will be run in concurrent mode, and all ORM write operations will be executed in blocking mode. This is expected to be both safe and fast, with our internal {\i SQLite3} engine, or most of the external databases. But you may need to change this default behavior, depending on the external engine you are connected to.
Most {\f1\fs20 TSQLDBConnectionProperties} will inherit from {\f1\fs20 TSQLDBConnectionPropertiesThreadSafe}, so will create one connection per thread. This is efficient, but some providers may have issues with it.
First of all, some database client libraries may not allow transactions to be shared among several threads - for instance @*MS SQL@. Other clients may consume a lot of resources for each connection, or may not have good multi-thread scaling abilities. Some database servers do fork their process for each connected client - for instance {\i @*PostgreSQL@}: you may want to reduce the server resources by using only one connection, so only one process on the server. To avoid such problems, you can force all ORM write operations to be executed in a dedicated thread, i.e. by setting {\f1\fs20 amMainThread} (which is not very opportune on a server without UI), or even better via {\f1\fs20 amBackgroundThread} or a {\f1\fs20 amBackgroundORMSharedThread}:
! aServer.AcquireExecutionMode[execORMWrite] := amBackgroundThread;
Secondly, especially on a long-running @*n-Tier@ {\i mORMot} server, you may suffer from broken connection exceptions. For instance, after a night without any activity, the attempts to access to the external database may fail in the morning, since the connection may have been disconnected by the database server in the meanwhile.\line You can use the {\f1\fs20 TSQLDBConnectionProperties.ConnectionTimeOutMinutes} property to specify a maximum period of inactivity after which all connections will be flushed and recreated, to avoid potential broken connections issues.\line In practice, recreating the connections after a while is safe and won't slow done the process - on the contrary, it may help reducing the consumed resources, and stabilize long running n-Tier servers.\line {\f1\fs20 ThreadSafeConnection} method will check for the last activity on its {\f1\fs20 TSQLDBConnectionProperties} instance, and then call {\f1\fs20 ClearConnectionPool} to release all active connections if the idle time elapsed was too long.
As a consequence, if you use this {\f1\fs20 ConnectionTimeOutMinutes} property, you should ensure that no other connection is still active on the background, otherwise some unexpected issues may occur.\line For instance, you should ensure that your {\i mORMot} ORM server runs all its statements in blocking mode for both read and write:
! aServer.AcquireExecutionMode[execORMGet] := am***;
! aServer.AcquireExecutionMode[execORMWrite] := am***;
Here above, safe blocking {\f1\fs20 am***} modes are any mode {\i but} {\f1\fs20 amUnlocked}, i.e. either {\f1\fs20 amLocked}, {\f1\fs20 amBackgroundThread}, {\f1\fs20 amBackgroundORMSharedThread} or {\f1\fs20 amMainThread}.
\page
:83External NoSQL database access
%cartoon06.png
Our @*ORM@ @*REST@ful framework is able to access not only regular @*SQL@ database engines, via @126@, but also @*NoSQL@ engines - see @82@.
Remember the diagram introducing {\i mORMot}'s @42@:
%%mORMotDBDesign
\page
The following {\i NoSQL} engines can be accessed from {\i mORMot}'s {\i Object Document Mapping} (ODM) abilities:
|%20%80
|\b NoSQL Engine|Description\b0
|{\f1\fs20 TObjectList}|In memory storage, with JSON or binary disk persistence
|{\i @*MongoDB@}|#1 NoSQL database engine
|%
We can in fact consider our {\f1\fs20 @**TSQLRestStorageInMemory@} instance, and its {\f1\fs20 TObjectList} storage, as a {\i NoSQL} very fast in-memory engine, written in pure Delphi. See @57@ for details about this feature.
{\i @**MongoDB@} (from "humongous") is a cross-platform document-oriented database system, and certainly the best known @*NoSQL@ database.\line According to @http://db-engines.com in December 2015, {\i MongoDB} is at 4th place of the most popular types of database management systems, and at first place for NoSQL database management systems.\line Our {\i mORMot} gives premium access to this database, featuring full @82@ abilities to the framework.
Integration is made at two levels:
- Direct low-level access to the {\i MongoDB} server, in the {\f1\fs20 SynMongoDB.pas} unit;
- Close integration with our ORM (which becomes {\i defacto} an @*ODM@), in the {\f1\fs20 mORMotMongoDB.pas} unit.
{\i MongoDB} eschews the traditional table-based relational database structure in favor of @*JSON@-like documents with dynamic schemas ({\i MongoDB} calls the format @*BSON@), which match perfectly {\i mORMot}'s @*REST@ful approach.
: SynMongoDB client
The {\f1\fs20 SynMongoDB.pas} unit features direct optimized access to a {\i MongoDB} server.
It gives access to any @**BSON@ data, including documents, arrays, and {\i MongoDB}'s custom types (like ObjectID, dates, binary, regex, Decimal128 or {\i Javascript}):
- For instance, a {\f1\fs20 TBSONObjectID} can be used to create some genuine document identifiers on the client side ({\i MongoDB} does not generate the IDs for you: a common way is to generate unique IDs on the client side);
- Generation of BSON content from any {\i Delphi} types (via {\f1\fs20 TBSONWriter});
- Fast in-place parsing of the BSON stream, without any memory allocation (via {\f1\fs20 TBSONElement});
- A {\f1\fs20 @*TBSONVariant@} custom variant type, to store {\i MongoDB}'s custom type values;
- Interaction with the {\f1\fs20 SynCommons.pas}' @80@ as document storage and late-binding access;
- Marshalling BSON to and from @*JSON@, with the {\i MongoDB} @*extended syntax@ for handling its custom types.
This unit defines some objects able to connect and manage databases and collections of documents on any {\i MongoDB} servers farm:
- Connection to one or several servers, including secondary hosts, via the {\f1\fs20 TMongoClient} class;
- Access to any database instance, via the {\f1\fs20 TMongoDatabase} class;
- Access to any collection, via the {\f1\fs20 TMongoCollection} class;
- It features some nice abilities about speed, like BULK insert or delete mode, and explicit {\i Write Concern} settings.
At collection level, you can have direct access to the data, with high level structures like {\f1\fs20 TDocVariant}/{\f1\fs20 TBSONVariant}, with easy-to-read JSON, or low level BSON content.\line You can also tune most aspects of the client process, e.g. about error handling or {\i write concerns} (i.e. how remote data modifications are acknowledged).
:  Connecting to a server
Here is some sample code, which is able to connect to a {\i MongoDB} server, and returns the server time:
!var Client: TMongoClient;
!    DB: TMongoDatabase;
!    serverTime: TDateTime;
!    res: variant; // we will return the command result as TDocVariant
!    errmsg: RawUTF8;
!begin
!  Client := TMongoClient.Create('localhost',27017);
!  try
!    DB := Client.Database['mydb'];
!    writeln('Connecting to ',DB.Name); // will write 'mydb'
!    errmsg := DB.RunCommand('hostInfo',res); // run a command
!    if errmsg<>'' then
!      exit; // quit on any error
!    serverTime := res.system.currentTime; // direct conversion to TDateTime
!    writeln('Server time is ',DateTimeToStr(serverTime));
!  finally
!    Client.Free; // will release the DB instance
!  end;
!end;
Note that for this low-level command, we used a {\f1\fs20 TDocVariant}, and its late-binding abilities.
In fact, if you put your mouse over the {\f1\fs20 res} variable during debugging, you will see the following JSON content:
µ{"system":{"currentTime":"2014-05-06T15:24:25","hostname":"Acer","cpuAddrSize":64,"memSizeMB":3934,"numCores":4,"cpuArch":"x86_64","numaEnabled":false},"os":{"type":"Windows","name":"Microsoft Windows 7","version":"6.1 SP1 (build 7601)"},"extra":{"pageSize":4096},"ok":1}
And we simply access to the server time by writing {\f1\fs20 res.system.currentTime}.
Here connection was made anonymously. It will work only if the {\f1\fs20 mongod} instance is running on the same computer. Safe remote connection, including user authentication, could be made via the {\f1\fs20 TMongoClient.OpenAuth()} method: it supports the latest {\f1\fs20 SCRAM-SHA-1} challenge-response mechanism (supported since {\i MongoDB} 3.x), or the deprecated {\f1\fs20 MONGODB-CR} (for older versions).
!    ...
!  Client := TMongoClient.Create('localhost',27017);
!  try
!    DB := Client.OpenAuth('mydb','mongouser','mongopwd');
!    ...
For safety reasons, never let a {\i MongoDB} server be remotely accessible without proper authentication, as stated by @http://docs.mongodb.org/manual/administration/security-access-control The {\f1\fs20 TMongoDatabase.CreateUser()}, {\f1\fs20 CreateUserForThisDatabase()} and {\f1\fs20 DropUser()} methods allow to easily manage credentials from your applications.
:  Adding some documents to the collection
We will now explain how to add documents to a given collection.
We assume that we have a {\f1\fs20 DB: TMongoDatabase} instance available. Then we will create the documents with a {\f1\fs20 TDocVariant} instance, which will be filled via late-binding, and via a {\f1\fs20 doc.Clear} pseudo-method used to flush any previous property value:
!var Coll: TMongoCollection;
!    doc: variant;
!    i: integer;
!begin
!  Coll := DB.CollectionOrCreate[COLL_NAME];
!  TDocVariant.New(doc);
!  for i := 1 to 10 do
!  begin
!    doc.Clear;
!    doc.Name := 'Name '+IntToStr(i+1);
!    doc.Number := i;
!!    Coll.Save(doc);
!    writeln('Inserted with _id=',doc._id);
!  end;
!end;
Thanks to {\f1\fs20 TDocVariant} late-binding abilities, code is pretty easy to understand and maintain.
This code will display the following on the console:
$Inserted with _id=5369029E4F901EE8114799D9
$Inserted with _id=5369029E4F901EE8114799DA
$Inserted with _id=5369029E4F901EE8114799DB
$Inserted with _id=5369029E4F901EE8114799DC
$Inserted with _id=5369029E4F901EE8114799DD
$Inserted with _id=5369029E4F901EE8114799DE
$Inserted with _id=5369029E4F901EE8114799DF
$Inserted with _id=5369029E4F901EE8114799E0
$Inserted with _id=5369029E4F901EE8114799E1
$Inserted with _id=5369029E4F901EE8114799E2
It means that the {\f1\fs20 Coll.Save()} method was clever enough to understand that the supplied document does not have any {\f1\fs20 _id} field, so will compute one on the client side before sending the document data to the {\i MongoDB} server.
We may have written:
!  for i := 1 to 10 do
!  begin
!    doc.Clear;
!!    doc._id := ObjectID;
!    doc.Name := 'Name '+IntToStr(i+1);
!    doc.Number := i;
!    Coll.Save(doc);
!    writeln('Inserted with _id=',doc._id);
!  end;
!end;
Which will compute the document identifier explicitly before calling {\f1\fs20 Coll.Save()}.\line In this case, we may have called directly {\f1\fs20 Coll.Insert()}, which is somewhat faster.
Note that you are not obliged to use a {\i MongoDB} ObjectID as identifier. You can use any value, if you are sure that it will be genuine. For instance, you can use an integer:
!  for i := 1 to 10 do
!  begin
!    doc.Clear;
!!    doc._id := i;
!    doc.Name := 'Name '+IntToStr(i+1);
!    doc.Number := i;
!!    Coll.Insert(doc);
!    writeln('Inserted with _id=',doc._id);
!  end;
!end;
The console will display now:
$Inserted with _id=1
$Inserted with _id=2
$Inserted with _id=3
$Inserted with _id=4
$Inserted with _id=5
$Inserted with _id=6
$Inserted with _id=7
$Inserted with _id=8
$Inserted with _id=9
$Inserted with _id=10
Note that the {\i mORMot} ORM will compute a genuine series of integers in a similar way, which will be used as expected by the {\f1\fs20 TSQLRecord.ID} @*primary key@ property.
The {\f1\fs20 TMongoCollection} class can also write a list of documents, and send them at once to the {\i MongoDB} server: this BULK insert mode - close to the {\i Array Binding} feature of some SQL providers, and implemented in our {\i SynDB.pas} classes - see @28@ - can increase the insertion by a factor of 10 times, even when connected to a local instance: imagine how much time it may save over a physical network!
For instance, you may write:
!var docs: TVariantDynArray;
!...
!  SetLength(docs,COLL_COUNT);
!  for i := 0 to COLL_COUNT-1 do begin
!    TDocVariant.New(docs[i]);
!    docs[i]._id := ObjectID; // compute new ObjectID on the client side
!    docs[i].Name := 'Name '+IntToStr(i+1);
!    docs[i].FirstName := 'FirstName '+IntToStr(i+COLL_COUNT);
!    docs[i].Number := i;
!  end;
!  Coll.Insert(docs); // insert all values at once
!...
You will find out later for some numbers about the speed increase due to such BULK insert.
:  Retrieving the documents
You can retrieve the document as a {\f1\fs20 TDocVariant} instance:
!var doc: variant;
!...
!  doc := Coll.FindOne(5);
!  writeln('Name: ',doc.Name);
!  writeln('Number: ',doc.Number);
Which will write on the console:
$Name: Name 6
$Number: 5
You have access to the whole {\f1\fs20 Query} parameter, if needed:
!  doc := Coll.FindDoc('{_id:?}',[5]);
!  doc := Coll.FindOne(5); // same as previous
This {\f1\fs20 Query} filter is similar to a WHERE clause in SQL. You can write complex search patterns, if needed - see @http://docs.mongodb.org/manual/reference/method/db.collection.find for reference.
You can retrieve a list of documents, as a dynamic array of {\f1\fs20 TDocVariant}:
!var docs: TVariantDynArray;
!...
!  Coll.FindDocs(docs);
!  for i := 0 to high(docs) do
!    writeln('Name: ',docs[i].Name,'  Number: ',docs[i].Number);
Which will output:
$Name: Name 2  Number: 1
$Name: Name 3  Number: 2
$Name: Name 4  Number: 3
$Name: Name 5  Number: 4
$Name: Name 6  Number: 5
$Name: Name 7  Number: 6
$Name: Name 8  Number: 7
$Name: Name 9  Number: 8
$Name: Name 10  Number: 9
$Name: Name 11  Number: 10
In a GUI application, you could fill a VCL grid using a {\f1\fs20 TDocVariantArrayDataSet} as defined in {\f1\fs20 SynVirtualDataSet.pas}, for instance:
! ds1.DataSet.Free; // release previous TDataSet
! ds1.DataSet := ToDataSet(self,FindDocs('{name:?,age:{$gt:?}}',['John',21],null));
This overloaded {\f1\fs20 FindDocs()} method takes a query filter as JSON and parameters (following the {\f1\fs20 MongoDB} syntax), and a {\f1\fs20 Projection} mapping ({\f1\fs20 null} to retrieve all properties). Its returns a {\f1\fs20 TVariantDynArray} result, which was mapped to an optimized read-only {\f1\fs20 TDataSet} using the overloaded {\f1\fs20 ToDataSet()} function. So in our case, the DB grid has been filled with all people named '{\f1\fs20 John}', with age greater than 21.
If you want to retrieve the documents directly as JSON, we can write:
!var json: RawUTF8;
!...
!  json := Coll.FindJSON(null,null);
!  writeln(json);
!...
This will append the following to the console:
$[{"_id":1,"Name":"Name 2","Number":1},{"_id":2,"Name":"Name 3","Number":2},{"_id
$":3,"Name":"Name 4","Number":3},{"_id":4,"Name":"Name 5","Number":4},{"_id":5,"N
$ame":"Name 6","Number":5},{"_id":6,"Name":"Name 7","Number":6},{"_id":7,"Name":"
$Name 8","Number":7},{"_id":8,"Name":"Name 9","Number":8},{"_id":9,"Name":"Name 1
$0","Number":9},{"_id":10,"Name":"Name 11","Number":10}]
You can note that {\f1\fs20 FindJSON()} has two properties, which are the {\f1\fs20 Query} filter, and a {\f1\fs20 Projection} mapping (similar to the column names in a {\f1\fs20 SELECT col1,col2}).\line So we may have written:
!  json := Coll.FindJSON('{_id:?}',[5]);
!  writeln(json);
Which outputs:
$[{"_id":5,"Name":"Name 6","Number":5}]
We used here an overloaded {\f1\fs20 FindJSON()} method, which accept the {\i MongoDB} @*extended syntax@ (here, the field name is unquoted), and parameters as variables.
We can specify a projection:
!  json := Coll.FindJSON('{_id:?}',[5],'{Name:1}');
!  writeln(json);
Which will only return the "Name" and "_id" fields (since {\f1\fs20 _id} is, by {\i MongoDB} convention, always returned:
$[{"_id":5,"Name":"Name 6"}]
To return only the "Name" field, you can specify {\f1\fs20 '{_id:0,Name:1}'} as JSON in @*extended syntax@ for the {\i projection} parameter.
$[{"Name":"Name 6"}]
There are other methods able to retrieve data, also directly as BSON binary data. They will be used for best speed e.g. in conjunction with our ORM, but for most end-user code, using {\f1\fs20 TDocVariant} is safer and easier to maintain.
:   Updating or deleting documents
The {\f1\fs20 TMongoCollection} class has some methods dedicated to alter existing documents.
At first, the {\f1\fs20 Save()} method can be used to update a document which has been first retrieved:
!  doc := Coll.FindOne(5);
!  doc.Name := 'New!';
!  Coll.Save(doc);
!  writeln('Name: ',Coll.FindOne(5).Name);
Which will write:
$Name: New!
Note that we used here an integer value (5) as key, but we may use an {\i ObjectID} instead, if needed.
The {\f1\fs20 Coll.Save()} method could be changed into {\f1\fs20 Coll.Update()}, which expects an explicit {\f1\fs20 Query} operator, in addition to the updated document content:
!  doc := Coll.FindOne(5);
!  doc.Name := 'New!';
!  Coll.Update(BSONVariant(['_id',5]),doc);
!  writeln('Name: ',Coll.FindOne(5).Name);
Note that by {\i MongoDB}'s design, any call to {\f1\fs20 Update()} will {\i replace} the whole document.
For instance, if you write:
!  writeln('Before: ',Coll.FindOne(3));
!  Coll.Update('{_id:?}',[3],'{Name:?}',['New Name!']);
!  writeln('After:  ',Coll.FindOne(3));
Then the {\f1\fs20 Number} field will disappear!
$Before: {"_id":3,"Name":"Name 4","Number":3}
$After:  {"_id":3,"Name":"New Name!"}
If you need to update only some fields, you will have to use the {\f1\fs20 $set} modifier:
!  writeln('Before: ',Coll.FindOne(4));
!  Coll.Update('{_id:?}',[4],'{$set:{Name:?}}',['New Name!']);
!  writeln('After:  ',Coll.FindOne(4));
Which will write on the console the value as expected:
$Before: {"_id":4,"Name":"Name 5","Number":4}
$After:  {"_id":4,"Name":"New Name!","Number":4}
Now the {\f1\fs20 Number} field remains untouched.
You can also use the {\f1\fs20 Coll.UpdateOne()} method, which will update the supplied fields, and leave the non specified fields untouched:
!  writeln('Before: ',Coll.FindOne(2));
!  Coll.UpdateOne(2,_Obj(['Name','NEW']));
!  writeln('After:  ',Coll.FindOne(2));
Which will output as expected:
$Before: {"_id":2,"Name":"Name 3","Number":2}
$After:  {"_id":2,"Name":"NEW","Number":2}
You can refer to the documentation of the {\f1\fs20 SynMongoDB.pas} unit, to find out all functions, classes and methods available to work with {\i MongoDB}.
Some very powerful features are available, including @**Aggregation@ (available since {\i MongoDB} 2.2), which offers a good alternative to standard @**Map/Reduce@ pattern.\line See @http://docs.mongodb.org/manual/reference/command/aggregate for reference.
:  Write Concern and Performance
You can take a look at the {\f1\fs20 MongoDBTests.dpr} sample - located in the {\f1\fs20 SQLite3\\Samples\\24 - MongoDB} sub-folder of the source code repository, and the {\f1\fs20 TTestDirect} classes, to find out some performance information.
In fact, this {\f1\fs20 TTestDirect} is inherited twice, to run the same tests with diverse write concern:
\graph HierTTestDirect MongoDB TTestDirect classes hierarchy
\TTestDirectWithAcknowledge\TTestDirect
\TTestDirectWithoutAcknowledge\TTestDirect
\
The difference between the two classes will take place at client initialization:
!procedure TTestDirect.ConnectToLocalServer;
!...
!  fClient := TMongoClient.Create('localhost',27017);
!  if ClassType=TTestDirectWithAcknowledge then
!!    fClient.WriteConcern := wcAcknowledged else
!  if ClassType=TTestDirectWithoutAcknowledge then
!!    fClient.WriteConcern := wcUnacknowledged;
!...
{\f1\fs20 wcAcknowledged} is the default safe mode: the {\i MongoDB} server confirms the receipt of the write operation. Acknowledged write concern allows clients to catch network, duplicate key, and other errors. But it adds an additional round-trip from the client to the server, and wait for the command to be finished before returning the error status: so it will slow down the write process.
With {\f1\fs20 wcUnacknowledged}, {\i MongoDB} does not acknowledge the receipt of write operation.  Unacknowledged is similar to errors ignored; however, drivers attempt to receive and handle network errors when possible. The driver's ability to detect network errors depends on the system's networking configuration.
The speed difference between the two is worth mentioning, as stated by the regression tests status, running on a local {\i MongoDB} instance:
$1. Direct access
$
$ 1.1. Direct with acknowledge:
$  - Connect to local server: 6 assertions passed  4.72ms
$  - Drop and prepare collection: 8 assertions passed  9.38ms
$!  - Fill collection: 15,003 assertions passed  558.79ms
$!     5000 rows inserted in 548.83ms i.e. 9110/s, aver. 109us, 3.1 MB/s
$  - Drop collection: no assertion  856us
$!  - Fill collection bulk: 2 assertions passed  74.59ms
$!     5000 rows inserted in 64.76ms i.e. 77204/s, aver. 12us, 7.2 MB/s
$  - Read collection: 30,003 assertions passed  2.75s
$     5000 rows read at once in 9.66ms i.e. 517330/s, aver. 1us, 39.8 MB/s
$!  - Update collection: 7,503 assertions passed  784.26ms
$!     5000 rows updated in 435.30ms i.e. 11486/s, aver. 87us, 3.7 MB/s
$!  - Delete some items: 4,002 assertions passed  370.57ms
$!     1000 rows deleted in 96.76ms i.e. 10334/s, aver. 96us, 2.2 MB/s
$  Total failed: 0 / 56,527  - Direct with acknowledge PASSED  4.56s
$
$ 1.2. Direct without acknowledge:
$  - Connect to local server: 6 assertions passed  1.30ms
$  - Drop and prepare collection: 8 assertions passed  8.59ms
$!  - Fill collection: 15,003 assertions passed  192.59ms
$!     5000 rows inserted in 168.50ms i.e. 29673/s, aver. 33us, 4.4 MB/s
$  - Drop collection: no assertion  845us
$!  - Fill collection bulk: 2 assertions passed  68.54ms
$!     5000 rows inserted in 58.67ms i.e. 85215/s, aver. 11us, 7.9 MB/s
$  - Read collection: 30,003 assertions passed  2.75s
$     5000 rows read at once in 9.99ms i.e. 500150/s, aver. 1us, 38.5 MB/s
$!  - Update collection: 7,503 assertions passed  446.48ms
$!     5000 rows updated in 96.27ms i.e. 51933/s, aver. 19us, 7.7 MB/s
$!  - Delete some items: 4,002 assertions passed  297.26ms
$!     1000 rows deleted in 19.16ms i.e. 52186/s, aver. 19us, 2.8 MB/s
$  Total failed: 0 / 56,527  - Direct without acknowledge PASSED  3.77s
As you can see, the reading speed is not affected by the {\i Write Concern} settings.\line But data writing can be multiple times faster, when each write command is not acknowledged.
Since there is no error handling, {\f1\fs20 wcUnacknowledged} is not to be used on production. You may use it for replication, or for data consolidation, e.g. feeding a database with a lot of existing data as fast as possible.
\page
:84 MongoDB + ORM = ODM
The {\f1\fs20 mORMotMongoDB.pas} unit is able to let any {\f1\fs20 TSQLRecord} class be persisted on a remote {\i MongoDB} server.
As a result, our @*ORM@ is able to be used as a @82@ framework, with almost no code change. Any {\i MongoDB} database can be accessed via @*REST@ful commands, using @*JSON@ over @*HTTP@ - see @6@.
This integration benefits from the other parts of the framework (e.g. our @*UTF-8@ dedicated process, which is also the native encoding for @*BSON@), so you can easily mix @*SQL@ and @*NoSQL@ databases with the exact same code, and are still able to tune any SQL or {\i MongoDB} request in your code, if necessary.
From the client point of view, there is no difference between a ORM or an @*ODM@: you may use a SQL engine as a storage for ODM - via @29@ - or even a NoSQL database as a regular ORM, with @*denormalization@ (even if it may void most advantages of NoSQL).
:  Define the TSQLRecord class
In the database model, we define a {\f1\fs20 TSQLRecord} class, as usual:
!  TSQLORM = class(TSQLRecord)
!  private
!    fAge: integer;
!    fName: RawUTF8;
!    fDate: TDateTime;
!    fValue: variant;
!    fInts: TIntegerDynArray;
!    fCreateTime: TCreateTime;
!    fData: TSQLRawBlob;
!  published
!    property Name: RawUTF8 read fName write fName stored AS_UNIQUE;
!    property Age: integer read fAge write fAge;
!    property Date: TDateTime read fDate write fDate;
!    property Value: variant read fValue write fValue;
!    property Ints: TIntegerDynArray index 1 read fInts write fInts;
!    property Data: TSQLRawBlob read fData write fData;
!    property CreateTime: TCreateTime read fCreateTime write fCreateTime;
!  end;
Note that we did not define any {\f1\fs20 {\b index} ...} values for the {\f1\fs20 RawUTF8} property, as we need for external SQL databases, since {\i MongoDB} does not expect any restriction about text fields length (as far as I know, the only SQL engines which allow this natively without any performance penalty are {\i SQlite3} and {\i @*PostgreSQL@}).
The property values will be stored in the native {\i MongoDB} layout, i.e. with a better coverage than the SQL types recognized by our {\f1\fs20 SynDB*} unit:
|%24%14%64
|\b Delphi|{\i MongoDB}|Remarks\b0
|{\f1\fs20 byte}|int32|
|{\f1\fs20 word}|int32|
|{\f1\fs20 integer}|int32|
|{\f1\fs20 cardinal}|N/A|You should use {\f1\fs20 Int64} instead
|{\f1\fs20 Int64}|int64|
|{\f1\fs20 boolean}|boolean|{\i MongoDB} has a {\f1\fs20 boolean} type
|enumeration|int32|store the ordinal value of the @*enumerated@ item(i.e. starting at 0 for the first element)
|set|int64|each bit corresponding to an enumerated item (therefore a set of up to 64 elements can be stored in such a field)
|{\f1\fs20 single}|{\f1\fs20 @*double@}|
|{\f1\fs20 double}|double|
|{\f1\fs20 extended}|double|stored as {\f1\fs20 double} (precision lost)
|{\f1\fs20 @*currency@}|double|stored as {\f1\fs20 double} ({\i MongoDB} does not have a BSD type)
|{\f1\fs20 @*RawUTF8@}|@*UTF-8@|this is the {\b preferred} field type for storing some textual content in the ORM
|{\f1\fs20 WinAnsiString}|UTF-8|{\i WinAnsi} char-set (code page 1252) in {\i Delphi}
|{\f1\fs20 RawUnicode}|UTF-8|{\i UCS2} char-set in {\i Delphi}, as {\f1\fs20 AnsiString}
|{\f1\fs20 @*WideString@}|UTF-8|{\i UCS2} char-set, as COM BSTR type (Unicode in all version of {\i Delphi})
|{\f1\fs20 @*SynUnicode@}|UTF-8|Will be either {\f1\fs20 WideString} before {\i Delphi} 2009, or {\f1\fs20 UnicodeString} later
|{\f1\fs20 string}|UTF-8|Not to be used before {\i Delphi} 2009 (unless you may loose some data during conversion) - {\f1\fs20 RawUTF8} is preferred in all cases
|{\f1\fs20 @*TDateTime@\line @*TDateTimeMS@}|datetime|@*ISO 8601@ encoded date time
|{\f1\fs20 @*TTimeLog@}|int64|as proprietary fast {\f1\fs20 Int64} date time
|{\f1\fs20 @*TModTime@}|int64|the server date time will be stored when a record is modified (as proprietary fast {\f1\fs20 Int64})
|{\f1\fs20 @*TCreateTime@}|int64|the server date time will be stored when a record is created (as proprietary fast {\f1\fs20 Int64})
|{\f1\fs20 @*TUnixTime@}|datetime|seconds since Unix epoch
|{\f1\fs20 @*TUnixMSTime@}|datetime|milliseconds since Unix epoch
|{\f1\fs20 @*TSQLRecord@}|int32|32-bit {\f1\fs20 RowID} pointing to another record (warning: the field value contains {\f1\fs20 pointer(RowID)}, not a valid object instance - the record content must be retrieved with late-binding via its {\f1\fs20 ID} using a {\f1\fs20 PtrInt(Field)} typecast or the {\f1\fs20 Field.ID} method), or by using e.g. {\f1\fs20 @*CreateJoined@()} - is 64-bit on {\i Win64}
|{\f1\fs20 @*TID@}|int32/int64|{\f1\fs20 RowID} pointing to another record - this kind of property is 64-bit compatible, so can handle values up to 9,223,372,036,854,775,808
|{\f1\fs20 @*TSQLRecordMany@}|nothing|data is stored in a separate {\i pivot} table; for MongoDB, you should better use {\i data sharding}, and an embedded sub-document
|{\f1\fs20 @*TRecordReference@}\line {\f1\fs20 @*TRecordReferenceToBeDeleted@}|int32/int64|store both {\f1\fs20 ID} and {\f1\fs20 TSQLRecord} type in a {\f1\fs20 @*RecordRef@}-like value - with proper synchronization when the record is deleted
|{\f1\fs20 @*TPersistent@}|object|@*BSON@ object (from {\f1\fs20 ObjectToJSON})
|{\f1\fs20 @*TCollection@}|array|BSON array of objects (from {\f1\fs20 ObjectToJSON})
|{\f1\fs20 @*TObjectList@}|array|BSON array of objects (from {\f1\fs20 ObjectToJSON}) - see {\f1\fs20 TJSONSerializer. @*RegisterClassForJSON@} @71@
|{\f1\fs20 @*TStrings@}|array|BSON array of strings (from {\f1\fs20 ObjectToJSON})
|{\f1\fs20 @*TRawUTF8List@}|array|BSON array of string (from {\f1\fs20 ObjectToJSON})
|any {\f1\fs20 @*TObject@}|object|See {\f1\fs20 TJSONSerializer.@*RegisterCustomSerializer@} @52@
|{\f1\fs20 @*TSQLRawBlob@}|binary|This type is an alias to {\f1\fs20 @*RawByteString@} - those properties are not retrieved by default: you need to use {\f1\fs20 RetrieveBlobFields()} or set {\f1\fs20 ForceBlobTransfert} / {\f1\fs20 ForceBlobTransertTable[]} properties
|{\f1\fs20 TByteDynArray}|binary|Used to embed a BLOB property stored as BSON binary within a document - so that {\f1\fs20 @*TSQLRawBlob@} may be restricted in the future to @*GridFS@ external content
|{\i @*dynamic array@s}|array\line binary|if the dynamic array can be saved as true JSON, will be stored as BSON array - otherwise, will be stored in the {\f1\fs20 TDynArray.SaveTo} BSON binary format
|{\f1\fs20 variant}|value\line array\line object|BSON number, text, date, object or array, depending on @80@ - or {\f1\fs20 TBSONVariant} stored value (e.g. to store native {\i MongoDB} types like {\f1\fs20 ObjectID} or {\f1\fs20 Decimal128})
|{\f1\fs20 record}|binary\line object|BSON as defined in code by overriding {\f1\fs20 TSQLRecord.InternalRegisterCustomProperties} to produce true JSON
|%
You can share the same {\f1\fs20 TSQLRecord} definition with {\i MongoDB} and other storage means, like external SQL databases. Unused information (like the {\f1\fs20 index} attribute) will just be ignored.
Note that {\f1\fs20 TSQLRecord}, {\f1\fs20 TID} and {\f1\fs20 TRecordReference*} published properties will automatically create an index on the corresponding field, and that a kind of {\f1\fs20 ON DELETE SET DEFAULT} tracking will take place for {\f1\fs20 TSQLRecord} and {\f1\fs20 TRecordReference} properties, and {\f1\fs20 ON DELETE CASCADE} for {\f1\fs20 TRecordReferenceToBeDeleted} - but not for {\f1\fs20 TID}, since we do not know which table to track.
:  Register the TSQLRecord class
On the server side (there won't be any difference for the client), you define a {\f1\fs20 TMongoDBClient}, and assign it to a given {\f1\fs20 TSQLRecord} class, via a call to {\f1\fs20 StaticMongoDBRegister()}:
!  MongoClient := TMongoClient.Create('localhost',27017);
!  DB := MongoClient.Database['dbname'];
!  Model := TSQLModel.Create([TSQLORM]);
!  Client := TSQLRestClientDB.Create(Model,nil,':memory:',TSQLRestServerDB);
!!  if StaticMongoDBRegister(TSQLORM,fClient.Server,fDB,'collectionname')=nil then
!    raise Exception.Create('Error');
And... that's all!
If all the tables of a {\i mORMot} server should be hosted on a {\i MongoDB} server, you could call the {\f1\fs20 @*StaticMongoDBRegisterAll@()} function instead:
! StaticMongoDBRegisterAll(aServer,aMongoClient.Open(colllectionname'));
If you want {\f1\fs20 TSQLRecord.InitializeTable} method to be called for void tables (and for instance create {\f1\fs20 TSQLAuthGroup} and {\f1\fs20 TSQLAuthUser} default content), you can execute the following command:
! Client.Server.InitializeTables(INITIALIZETABLE_NOINDEX);
You can then execute any ORM command, as usual:
!  writeln(Client.TableRowCount(TSQLORM)=0);
As with external databases, you can specify the field names mapping between the objects and the {\i MongoDB} collection.\line By default, the {\f1\fs20 TSQLRecord.ID} property is mapped to the {\i MongoDB}'s {\f1\fs20 _id} field, and the ORM will populate this {\f1\fs20 _id} field with a sequence of integer values, just like any {\f1\fs20 TSQLRecord} table.\line You can specify your own mapping, using e.g.:
! aModel.Props[aClass].ExternalDB.MapField(..)
Since the field names are stored within the document itself, it may be a good idea to use shorter naming for the {\i MongoDB} collection. It may save some storage space, when working with a huge number of documents.
Once the {\f1\fs20 TSQLRecord} is mapped to a {\i MongoDB} collection, you can always have direct access to the corresponding {\f1\fs20 TMongoCollection} instance later on, using a simple transtyping:
! (aServer.StaticDataServer[aClass] as TSQLRestStorageMongoDB).Collection
This may allow any specific task, including any tuned query or process.
:  ORM/ODM CRUD methods
You can add documents with the standard CRUD methods of the ORM, as usual:
!  R := TSQLORM.Create;
!  try
!    for i := 1 to COLL_COUNT do begin
!      R.Name := 'Name '+Int32ToUTF8(i);
!      R.Age := i;
!      R.Date := 1.0*(30000+i);
!      R.Value := _ObjFast(['num',i]);
!      R.Ints := nil;
!      R.DynArray(1).Add(i);
!!      assert(Client.Add(R,True)=i);
!    end;
!  finally
!    R.Free;
!  end;
As we already saw, the framework is able to handle any kind of properties, including complex types like {\i @*dynamic array@s} or {\f1\fs20 variant}.\line In the above code, a {\f1\fs20 TDocVariant} document has been stored in {\f1\fs20 R.Value}, and a dynamic array of {\f1\fs20 integer} values is accessed via its {\f1\fs20 @*index@ 1} shortcut and the {\f1\fs20 TSQLRecord.DynArray()} method.
The usual {\f1\fs20 Retrieve} / {\f1\fs20 Delete} / {\f1\fs20 Update} methods are available:
!  R := TSQLORM.Create;
!  try
!    for i := 1 to COLL_COUNT do begin
!!      Check(Client.Retrieve(i,R));
!      // here R instance contains all values of one document, excluding BLOBs
!    end;
!  finally
!    R.Free;
!  end;
You can define a WHERE clause, as if the back-end where a regular SQL database:
!    R := TSQLORM.CreateAndFillPrepare(Client,'ID=?',[i]);
!    try
!    ...
:  ODM complex queries
To perform a query and retrieve the content of several documents, you can use regular {\f1\fs20 CreateAndFillPrepare} or {\f1\fs20 FillPrepare} methods:
!!  R := TSQLORM.CreateAndFillPrepare(Client,WHERE_CLAUSE,[WHERE_PARAMETERS]);
!  try
!    n := 0;
!!    while R.FillOne do begin
!      // here R instance contains all values of one document, excluding BLOBs
!      inc(n);
!    end;
!    assert(n=COLL_COUNT);
!  finally
!    R.Free;
!  end;
A WHERE clause can also be defined for {\f1\fs20 CreateAndFillPrepare} or {\f1\fs20 FillPrepare} methods. This WHERE clause could contain several expressions, joined with {\f1\fs20 AND} / {\f1\fs20 OR}.\line Each of those expressions could use:
- The simple comparators {\f1\fs20 = < <= <> > >=},
- An {\f1\fs20 IN (....)} clause,
- {\f1\fs20 IS NULL} / {\f1\fs20 IS NOT NULL} tests,
- A {\f1\fs20 LIKE} operation,
- Or even any {\f1\fs20 ...DynArrayContains()} specific function.
The {\i mORMot} ODM will convert this SQL-like statement into the optimized {\f1\fs20 MongoDB} query expression, using e.g. a regular expression for the {\f1\fs20 LIKE} operator.
The {\f1\fs20 LIMIT}, {\f1\fs20 OFFSET} and {\f1\fs20 ORDER BY} clauses will also be handled as expected. A special care should be taken for an {\f1\fs20 ORDER BY} on textual values: by design, {\i MongoDB} will always sort text with case-sensitivity, which is not what we expect: so our ODM will sort such content on client side, after having been retrieved from the {\i MongoDB} server. For numerical fields, {\i MongoDB} sorting features will be processed on the server side.
The {\f1\fs20 COUNT(*)} function will also be converted into the proper {\i MongoDB} API call, so that such operations will be as costless as possible. {\f1\fs20 DISTINCT() MAX() MIN() SUM() AVG()} functions and the {\f1\fs20 GROUP BY} clause will also be converted into optimized {\i MongoDB} aggregation pipelines, on the fly. You could even set aliases for the columns (e.g. {\f1\fs20 max(RowID) as first}) and perform simple addition/substraction of an integer value.
Here are some typical WHERE clauses, and the corresponding {\i MongoDB} query document as generated by the ODM:
|%50%50
|\b WHERE clause|{\i MongoDB} Query\b0
|{\f1\fs20 'Name=?',['Name 43']}|{\f1\fs20 \{Name:"Name 43"\}}
|{\f1\fs20 'Age<?',[51]}|{\f1\fs20 \{Age:\{$lt:51\}\}}
|{\f1\fs20 'Age in (1,10,20)'}|{\f1\fs20 \{Age:\{$in:[1,10,20]\}\}}
|{\f1\fs20 'Age in (1,10,20) and ID=?',[10]}|{\f1\fs20 \{Age:\{$in:[1,10,20]\},_id:10\}}
|{\f1\fs20 'Age in (10,20) or ID=?',[30]}|{\f1\fs20 \{$or:[\{Age:\{$in:[10,20]\}\},\{_id:30\}]\}}
|{\f1\fs20 'Name like ?',['name 1%']}|{\f1\fs20 \{Name:/^name 1/i\}}
|{\f1\fs20 'Name like ?',['name 1']}|{\f1\fs20 \{Name:/^name 1$/i\}}
|{\f1\fs20 'Name like ?',['%ame 1%']}|{\f1\fs20 \{Name:/ame 1/i\}}
|{\f1\fs20 'Data is null'}|{\f1\fs20 \{Data:null\}}
|{\f1\fs20 'Data is not null'}|{\f1\fs20 \{Data:\{$ne:null\}\}}
|{\f1\fs20 'Age<? limit 10',[51]}|{\f1\fs20 \{Age:\{$lt:51\}\}} + limit 10
|{\f1\fs20 'Age in (10,20) or ID=? order by ID desc',[30]}|{\f1\fs20 \{$query:\{$or:[\{Age:\{$in:[10,20]\}\},\{_id:30\}]\},$orderby:\{_id:-1\}\}}
|{\f1\fs20 'order by Name'}|{\f1\fs20 \{\}} + client side text sort by {\f1\fs20 Name}
|{\f1\fs20 'Age in (1,10,20) and IntegerDynArrayContains(Ints,?)',[10])}|{\f1\fs20 \{Age:\{$in:[1,10,20]\},Ints:\{$in:[10]\}\}}
|{\f1\fs20 Distinct(Age),max(RowID) as first,count(Age) as count\line group by age}|{\f1\fs20 \{$group:\{_id:"$Age",f1:\{$max:"$_id"\},f2:\{$sum:1\}\}\},\{$project:\{_id:0,"Age":"$_id","first":"$f1","count":"$f2"\}\}}
|{\f1\fs20 min(RowID),max(RowID),Count(RowID)}|{\f1\fs20 \{$group:\{_id:null,f0:\{$min:"$_id"\},f1:\{$max:"$_id"\},f2:\{$sum:1\}\}\},\{$project:\{_id:0,"min(RowID)":"$f0","max(RowID)":"$f1","Count(RowID)":"$f2"\}\}}
|{\f1\fs20 min(RowID) as a,max(RowID)+1 as b,Count(RowID) as c}|{\f1\fs20 \{$group:\{_id:null,f0:\{$min:"$_id"\},f1:\{$max:"$_id"\},f2:\{$sum:1\}\}\},\{$project:\{_id:0,"a":"$f0","b":\{$add:["$f1",1]\},"c":"$f2"\}\}}
|%
Note that parenthesis and mixed {\f1\fs20 AND} {\f1\fs20 OR} expressions are not handled yet. You could always execute any complex {\i NoSQL} query (e.g. using aggregation functions or the {\i @*Map/Reduce@} pattern) by using directly the {\f1\fs20 TMongoCollection} methods.
But for most business code, {\i mORMot} allows to share the same exact code between your regular SQL databases or NoSQL engines. You do not need to learn the {\i MongoDB} query syntax: the ODM will compute the right expression for you, depending on the database engine it runs on.
:  BATCH mode
In addition to individual @*CRUD@ operations, our {\i MongoDB} is able to use BATCH mode for adding or deleting documents.
You can write the exact same code as with any SQL back-end:
!  Client.BatchStart(TSQLORM);
!  R := TSQLORM.Create;
!  try
!    for i := 1 to COLL_COUNT do begin
!      R.Name := 'Name '+Int32ToUTF8(i);
!      R.Age := i;
!      R.Date := 1.0*(30000+i);
!      R.Value := _ObjFast(['num',i]);
!      R.Ints := nil;
!      R.DynArray(1).Add(i);
!      assert(Client.BatchAdd(R,True)>=0);
!    end;
!  finally
!    R.Free;
!  end;
!  assert(Client.BatchSend(IDs)=HTTP_SUCCESS);
Or for deletion:
!  Client.BatchStart(TSQLORM);
!  for i := 5 to COLL_COUNT do
!    if i mod 5=0 then
!      assert(fClient.BatchDelete(i)>=0);
!  assert(Client.BatchSend(IDs)=HTTP_SUCCESS);
Speed benefit may be huge in regard to individual Add/Delete operations, even on a local {\i MongoDB} server. We will see some benchmark numbers now.
:  ORM/ODM performance
You can take a look at @59@ to compare {\i MongoDB} as back-end for our ORM classes.
In respect to external @*SQL@ engines, it features very high speed, low CPU use, and almost no difference in use. We interfaced the {\f1\fs20 BatchAdd()} and {\f1\fs20 BatchDelete()} methods to benefit of {\i MongoDB} BULK process, and avoided most memory allocation during the process.
Here are some numbers, extracted from the {\f1\fs20 MongoDBTests.dpr} sample, which reflects the performance of our ORM/ODM, depending on the {\i Write Concern} mode used:
$2. ORM
$
$ 2.1. ORM with acknowledge:
$  - Connect to local server: 6 assertions passed  18.65ms
$  - Insert: 5,002 assertions passed  521.25ms
$!     5000 rows inserted in 520.65ms i.e. 9603/s, aver. 104us, 2.9 MB/s
$  - Insert in batch mode: 5,004 assertions passed  65.37ms
$!     5000 rows inserted in 65.07ms i.e. 76836/s, aver. 13us, 8.4 MB/s
$  - Retrieve: 45,001 assertions passed  640.95ms
$     5000 rows retrieved in 640.75ms i.e. 7803/s, aver. 128us, 2.1 MB/s
$  - Retrieve all: 40,001 assertions passed  20.79ms
$!     5000 rows retrieved in 20.33ms i.e. 245941/s, aver. 4us, 27.1 MB/s
$  - Retrieve one with where clause: 45,410 assertions passed  673.01ms
$     5000 rows retrieved in 667.17ms i.e. 7494/s, aver. 133us, 2.0 MB/s
$  - Update: 40,002 assertions passed  681.31ms
$     5000 rows updated in 660.85ms i.e. 7565/s, aver. 132us, 2.4 MB/s
$  - Blobs: 125,003 assertions passed  2.16s
$     5000 rows updated in 525.97ms i.e. 9506/s, aver. 105us, 2.4 MB/s
$  - Delete: 38,003 assertions passed  175.86ms
$     1000 rows deleted in 91.37ms i.e. 10944/s, aver. 91us, 2.3 MB/s
$  - Delete in batch mode: 33,003 assertions passed  34.71ms
$!     1000 rows deleted in 14.90ms i.e. 67078/s, aver. 14us, 597 KB/s
$  Total failed: 0 / 376,435  - ORM with acknowledge PASSED  5.00s
$
$ 2.2. ORM without acknowledge:
$  - Connect to local server: 6 assertions passed  16.83ms
$  - Insert: 5,002 assertions passed  179.79ms
$!     5000 rows inserted in 179.15ms i.e. 27908/s, aver. 35us, 3.9 MB/s
$  - Insert in batch mode: 5,004 assertions passed  66.30ms
$!     5000 rows inserted in 31.46ms i.e. 158891/s, aver. 6us, 17.5 MB/s
$  - Retrieve: 45,001 assertions passed  642.05ms
$     5000 rows retrieved in 641.85ms i.e. 7789/s, aver. 128us, 2.1 MB/s
$  - Retrieve all: 40,001 assertions passed  20.68ms
$!     5000 rows retrieved in 20.26ms i.e. 246718/s, aver. 4us, 27.2 MB/s
$  - Retrieve one with where clause: 45,410 assertions passed  680.99ms
$     5000 rows retrieved in 675.24ms i.e. 7404/s, aver. 135us, 2.0 MB/s
$  - Update: 40,002 assertions passed  231.75ms
$     5000 rows updated in 193.74ms i.e. 25807/s, aver. 38us, 3.6 MB/s
$  - Blobs: 125,003 assertions passed  1.44s
$     5000 rows updated in 150.58ms i.e. 33202/s, aver. 30us, 2.6 MB/s
$  - Delete: 38,003 assertions passed  103.57ms
$     1000 rows deleted in 19.73ms i.e. 50668/s, aver. 19us, 2.4 MB/s
$  - Delete in batch mode: 33,003 assertions passed  47.50ms
$!     1000 rows deleted in 364us i.e. 2747252/s, aver. 0us, 23.4 MB/s
$  Total failed: 0 / 376,435  - ORM without acknowledge PASSED  3.44s
As for direct {\i MongoDB} access, the {\f1\fs20 wcUnacknowledged} is not to be used on production, but may be very useful in some particular scenarios. As expected, the reading process is not impacted by the {\i Write Concern} mode set.
:6JSON RESTful Client-Server
%cartoon07.png
Before describing the Client-Server design of this framework, we may have to detail some standards it is based on:
- JSON as its internal data storage and transmission format;
- REST as its Client-Server architecture.
:2 JSON
:  Why use JSON?
As we just stated, the @**JSON@ format is used internally in this framework. By definition, the {\i JavaScript Object Notation} (JSON) is a standard, open and lightweight computer data interchange format.
JSON's basic types are - as retrieved from @http://en.wikipedia.org/wiki/JSON
|%15%85
|\b Type|Description\b0
|Number|{\f1\fs20 @*Double@} precision floating-point format in {\i JavaScript}, generally depends on implementation. There is no specific {\f1\fs20 integer} type
|String|Double-quoted Unicode, with backslash escaping
|Boolean|{\f1\fs20 true} or {\f1\fs20 false}
|Array|An ordered sequence of values, comma-separated and enclosed in square brackets; the values do not need to be of the same type
|Object|An unordered collection of {\f1\fs20 key:value} pairs with the '{\f1\fs20 :}' character separating the key and the value, comma-separated and enclosed in curly braces; the keys must be strings and should be distinct from each other
|{\f1\fs20 null}|Empty/undefined value
|%
Non-significant white space may be added freely around the "structural characters" (i.e. brackets "{\f1\fs20 \{ \} [ ]}", colons "{\f1\fs20 :}" and commas "{\f1\fs20 ,}").
The following example shows the JSON representation of an object that describes a person.\line The object has string fields for first name and last name, a number field for age, an object representing the person's address and an array of phone number objects.
${
$    "firstName": "John",
$    "lastName": "Smith",
$    "age": 25,
$    "address": {
$        "streetAddress": "21 2nd Street",
$        "city": "New York",
$        "state": "NY",
$        "postalCode": 10021
$    },
$    "phoneNumbers": [
$        {
$            "type": "home",
$            "number": "212 555-1234"
$        },
$        {
$            "type": "fax",
$            "number": "646 555-4567"
$        };
$    ]
$}
Usage of this layout, instead of other like XML or any proprietary format, results in several particularities:
- Like XML, it is a text-based, human-readable format for representing simple data structures and associative arrays (called objects);
- It's easier to read (for both human beings and machines), quicker to implement, and much smaller in size than XML for most use;
- It's a very efficient format for data caching;
- Its layout allows to be rewritten in place into individual zero-terminated @*UTF-8@ strings, with almost no wasted space: this feature is used for very fast JSON to text conversion of the tables results, with no memory allocation nor data copy;
- It's natively supported by the {\i @*JavaScript@} language, making it a perfect @*serialization@ format in any @*AJAX@ (i.e. Web 2.0) or HTML5 Mobile application;
- The JSON format is simple, and specified in a short and clean RFC document;
- The default text encoding for both JSON and {\i @*SQLite3@} is UTF-8, which allows the full Unicode char-set to be stored and communicated;
- It is the default data format used by ASP.NET AJAX services created in Windows Communication Foundation (WCF) since .NET framework 3.5; so it's Microsoft officially "ready";
- For binary @*BLOB@ transmission, we simply encode the binary data as {\i @*Base64@}; please note that, by default, BLOB fields are not transmitted over REST with other fields in JSON objects, see @1@ (only exception are {\i @*dynamic array@} fields, which are transmittest within the other fields).
REST JSON @**serialization@ will indeed be used in our main @*ORM@ to process of any {\f1\fs20 TSQLRecord} published properties, and in the {\f1\fs20 interface}-based @*SOA@ architecture of the framework, for content transmission.
In the framework, the whole @http://json.org standard is implemented, with some exceptions/extensions:
- {\f1\fs20 #0} characters will indicate the end of input, as with almost all JSON libraries - so if your text input contains a {\f1\fs20 #0} char, please handle it as binary (note that other control chars are escaped as expected);
- You may use an "@**extended syntax@" (used e.g. by {\f1\fs20 @*MongoDB@}) by unquoting ASCII-only property names;
- Floating point numbers are sometimes limited to {\f1\fs20 @*currency@} (i.e. 4 decimals), to ensure serialization/unserialization won't loose precision; but in such cases, it can be extended to the {\f1\fs20 double} precision via a set of options;
- There is no @*53-bit@ limitation for integers, as with @*JavaScript@: the framework handle 64-bit integer values - when using a JavaScript back-end, you may have to transmit huge values as text.
In practice, JSON has been found out to be very easy to work with and stable. A binary format is not used for transmission yet, but is available at other level of the framework, e.g. as an possible file format for in-memory {\f1\fs20 TObjectList} database engine (with our @*SynLZ@ compression - see @20@).
:  Values serialization
Standard {\i Delphi} value types are serialized directly within the JSON content, in their textual representation. For instance, {\f1\fs20 integer} or {\f1\fs20 Int64} are stored as numbers, and {\f1\fs20 @*double@} values are stored as their corresponding floating-point representation.
All {\f1\fs20 string} content is serialized as standard JSON text field, i.e. nested with double quotes ({\f1\fs20 "}). Since JSON uses @*UTF-8@ encoding, it is one of the reasons why we introduced the {\f1\fs20 @*RawUTF8@} type, and use it everywhere in our framework.
:51  Record serialization
In {\i Delphi}, the {\f1\fs20 @**record@} has some nice advantages:
- {\f1\fs20 record} are {\i value} objects, i.e. accessed by value, not by reference - this can be very convenient, e.g. when defining @54@;
- {\f1\fs20 record} can contain any other {\f1\fs20 record} or {\i dynamic array}, so are very convenient to work with (no need to define sub-classes or lists);
- {\f1\fs20 record} variables can be allocated on stack, so won't solicited the global heap;
- {\f1\fs20 record} instances automatically freed by the compiler when they come out of scope, so you won't need to write any {\f1\fs20 try..finally Free; end} block.
Serialization of {\f1\fs20 record} values are therefore a must-have for a framework like {\i mORMot}. In practice, the {\f1\fs20 record} types should be defined as {\f1\fs20 packed record}, so that low-level access will be easier to manage by the serializers.
:   Automatic serialization via Enhanced RTTI
Since {\i Delphi} 2010, the compiler generates additional RTTI at compilation, so that all {\f1\fs20 record} fields are described, and available at runtime.\line By the way, this @**enhanced RTTI@ is one of the reasons why executables did grow so much in newer versions of the compiler.
Our {\f1\fs20 SynCommons.pas} unit is able to use this enhanced information, and let any {\f1\fs20 record} be serialized via {\f1\fs20 RecordLoad()} and {\f1\fs20 RecordSave()} functions, and all internal JSON marshalling process.
In short, you have nothing to do. Just use your {\f1\fs20 record} as parameters, and, with {\i Delphi} 2010 and up, they will be serialized as valid JSON objects. The only restriction is that the records should be defined as {\f1\fs20 packed record}.
:   Serialization for older Delphi versions
Sadly, the information needed to serialize a {\f1\fs20 record} is available only since {\i Delphi} 2010.
If your application is developped on any older revision (e.g. {\i Delphi} 7, {\i Delphi} 2007 or {\i Delphi} 2009), you won't be able to automatically serialize {\f1\fs20 records} as plain JSON objects directly.
You have several paths available:
- By default, the {\f1\fs20 record} will be serialized as binary, and encoded as {\i Base64} text;
- Or you can define method callbacks which will write or read the data as you expect;
- Or you can define the {\f1\fs20 record} layout as plain text.
Note that any custom serialization (either via callbacks, or via text definition), will override any previous registered method, even the mechanism using the enhanced RTTI. You can change the default serialization to easily meet your requirements. For instance, this is what {\f1\fs20 SynCommons.pas} does for any {\f1\fs20 TGUID} content, which is serialized as the standard JSON text layout (e.g.  {\f1\fs20 "C9A646D3-9C61-4CB7-BFCD-EE2522C8F633"}), and not following the {\f1\fs20 TGUID record} layout as defined in the RTTI , i.e. {\f1\fs20 \{"D1":12345678,"D2":23023,"D3":9323,"D4":"0123456789ABCDEF"\}} - which is far from convenient.
:    Default Binary/Base64 serialization
On any version of the compiler prior to {\i Delphi} 2010, any {\f1\fs20 @*record@} value will be serialized by default with a proprietary binary (and optimized) layout - i.e. via {\f1\fs20 @*RecordLoad@} and {\f1\fs20 @*RecordSave@} functions - then encoded as {\i @**Base64@}, to be stored as plain text within the JSON stream.
A special @*UTF-8@ prefix (which does not match any existing {\i Unicode} glyph) is added at the beginning of the resulting JSON string to identify this content as a BLOB, as such:
$ { "MyRecord": "ï¿°w6nDoMOnYQ==" }
You will find in {\f1\fs20 SynCommons.pas} unit both {\f1\fs20 BinToBase64} and {\f1\fs20 Base64ToBin} functions, very optimized for speed. {\i Base64} encoding was chosen since it is standard, much more efficient than hexadecimal, and still JSON compatible without the need to escape its content.
When working with most part of the framework, you do not have anything to do: any record will by default follow this {\i Base64} serialization, so you will be able e.g. to publish or consume interface-based services with records.
:    Custom serialization
{\i Base64} encoding is pretty convenient for a computer (it is a compact and efficient format), but it is very limited about its interoperability. Our format is proprietary, and will use the internal {\i Delphi} serialization scheme: it means that it won't be readable nor writable outside the scope of your own {\i mORMot} applications. In a @*REST@ful/@*SOA@ world, this sounds not like a feature, but a limitation.
Custom {\f1\fs20 record} @*JSON@ serialization can therefore be defined, as with any {\f1\fs20 class} - see @52@. It will allow writing and parsing {\f1\fs20 record} variables as regular JSON objects, ready to be consumed by any client or server. Internally, some callbacks will be used to perform the serialization.
In fact, there are two entry points to specify a custom JSON serialization for {\f1\fs20 record}:
- When setting a custom {\i dynamic array} JSON serializer - see @53@ - the associated {\f1\fs20 record} will also use the same {\f1\fs20 Reader} and {\f1\fs20 Writer} callbacks;
- By setting explicitly serialization callbacks for the {\f1\fs20 TypeInfo()} of the record, with the very same {\f1\fs20 TTextWriter. @*RegisterCustomJSONSerializer@} method used for dynamic arrays.
Then the {\f1\fs20 Reader} and {\f1\fs20 Writer} callbacks can be defined by two means:
- By hand, i.e. coding the methods with manual conversion to JSON text or parsing;
- Via some text-based type definition, which will follow the {\f1\fs20 record} layout, but will do all the marshalling (including memory allocation) on its own.
:    Defining callbacks
For instance, if you want to serialize the following {\f1\fs20 record}:
!  TSQLRestCacheEntryValue = record
!    ID: TID;
!    Timestamp: cardinal;
!    JSON: RawUTF8;
!  end;
With the following code:
!  TTextWriter.RegisterCustomJSONSerializer(TypeInfo(TSQLRestCacheEntryValue),
!    TTestServiceOrientedArchitecture.CustomReader,
!    TTestServiceOrientedArchitecture.CustomWriter);
The expected format will be as such:
& {"ID":1786554763,"Timestamp":323618765,"JSON":"D:\\TestSQL3.exe"}
Therefore, the writer callback could be:
!class procedure TTestServiceOrientedArchitecture.CustomWriter(
!  const aWriter: TTextWriter; const aValue);
!var V: TSQLRestCacheEntryValue absolute aValue;
!begin
!  aWriter.AddJSONEscape(['ID',V.ID,'Timestamp',Int64(V.Timestamp),'JSON',V.JSON]);
!end;
In the above code, the {\f1\fs20 cardinal} field named {\f1\fs20 Timestamp} is type-casted to a {\f1\fs20 Int64}: in fact, as stated by the documentation of the {\f1\fs20 AddJSONEscape} method, an {\f1\fs20 array of const} will handle by default any {\f1\fs20 cardinal} as an {\f1\fs20 integer} value (this is a limitation of the {\i Delphi} compiler). By forcing the type to be an {\f1\fs20 Int64}, the expected {\f1\fs20 cardinal} value will be transmitted, and not a wrongly negative versions for numbers {\f1\fs20 > $7fffffff}.
On the other side, the corresponding reader callback will be like:
!class function TTestServiceOrientedArchitecture.CustomReader(P: PUTF8Char;
!  var aValue; out aValid: Boolean; aCustomVariantOptions: PDocVariantOptions): PUTF8Char;
!var V: TSQLRestCacheEntryValue absolute aValue;
!    Values: array[0..2] of TValuePUTF8Char;
!begin
!  result := JSONDecode(P,['ID','Timestamp','JSON'],@Values);
!  if result=nil then
!    aValid := false else begin
!    V.ID := GetInt64(Values[0].Value);
!    V.Timestamp := GetCardinal(Values[1].Value);
!    Values[2].ToUTF8(V.JSON);
!    aValid := true;
!  end;
!end;
Here {\f1\fs20 JSONDecode()} is used for fast deserialization of a JSON object.
:    Text-based definition
Writing those callbacks by hand could be error-prone, especially for the {\f1\fs20 Reader} event.
You can use the {\f1\fs20 TTextWriter.@*RegisterCustomJSONSerializerFromText@} method to define the {\f1\fs20 record} layout in a convenient text-based format. Once more, those types need to be defined as {\f1\fs20 packed record}, so that the text layout definition will not depend on compiler-specific field alignment.
The very same {\f1\fs20 TSQLRestCacheEntryValue} can be defined as with a typical {\i pascal} {\f1\fs20 record}:
! const
!  __TSQLRestCacheEntryValue = 'ID: Int64; Timestamp: cardinal; JSON: RawUTF8';
Or with a shorter syntax:
! const
!  __TSQLRestCacheEntryValue = 'ID Int64 Timestamp cardinal JSON RawUTF8';
Both declarations will do the same definition. Note that the supplied text should match {\i exactly} the original {\f1\fs20 record} type definition: do not swap or forget any property!
By convention, we use two underscore characters ({\f1\fs20 __}) before the {\f1\fs20 record} type name, to easily identify the layout definition. It may indeed be convenient to write it as a constant, close to the {\f1\fs20 record} type definition itself, and not in-lined at {\f1\fs20 RegisterCustomJSONSerializerFromText()} call level.
Then you register your type as such:
!  TTextWriter.RegisterCustomJSONSerializerFromText(
!    TypeInfo(TSQLRestCacheEntryValue),__TSQLRestCacheEntryValue);
Now you are able to serialize any {\f1\fs20 record} value directly:
!  Cache.ID := 10;
!  Cache.Timestamp := 200;
!  Cache.JSON := 'test';
!!  U := RecordSaveJSON(Cache,TypeInfo(TSQLRestCacheEntryValue));
!  Check(U='{"ID":10,"Timestamp":200,"JSON":"test"}');
You can also unserialize some existing JSON content:
!  U := '{"ID":210,"Timestamp":2200,"JSON":"test2"}';
!!  RecordLoadJSON(Cache,@U[1],TypeInfo(TSQLRestCacheEntryValue));
!  Check(Cache.ID=210);
!  Check(Cache.Timestamp=2200);
!  Check(Cache.JSON='test2');
Note that this text-based definition is very powerful, and is able to handle any level of nested {\f1\fs20 record} or {\i dynamic arrays}.
By default, it will write the JSON content in a compact form, and will expect only existing fields to be available in the incoming JSON. You can specify some options at registration, to ignore all non defined fields. It can be very useful when you want to consume some remote service, and are interested only in a few fields.
For instance, we may define a client access to a RESTful service like {\f1\fs20 api.github.com}:
!type
!  TTestCustomJSONGitHub = packed record
!    name: RawUTF8;
!    id: cardinal;
!    description: RawUTF8;
!    fork: boolean;
!    owner: record
!      login: RawUTF8;
!      id: cardinal;
!    end;
!  end;
!  TTestCustomJSONGitHubs = array of TTestCustomJSONGitHub;
!
!const
!  __TTestCustomJSONGitHub = 'name RawUTF8 id cardinal description RawUTF8 '+
!    'fork boolean owner{login RawUTF8 id cardinal}';
Note the {\f1\fs20 \{ \}} format to define a nested record, as a shorter alternative to a nested {\f1\fs20 record .. end} syntax.
It is also mandatory that you declare the {\f1\fs20 record} as {\f1\fs20 packed}. Otherwise, you may have unexpected access violation issues, since alignement may vary, depending on local setting, and compiler revision.
Now we can register the {\f1\fs20 record} layout, and provide some additional options:
!  TTextWriter.RegisterCustomJSONSerializerFromText(TypeInfo(TTestCustomJSONGitHub),
!    __TTestCustomJSONGitHub).Options := [soReadIgnoreUnknownFields,soWriteHumanReadable];
Here, we defined:
- {\f1\fs20 soReadIgnoreUnknownFields} to ignore any non defined field in the incoming JSON;
- {\f1\fs20 soWriteHumanReadable} to let the output JSON be more readable.
Then the JSON can be parsed then emitted as such:
!var git: TTestCustomJSONGitHubs;
! ...
!  U := zendframeworkJson;
!!  Check(DynArrayLoadJSON(git,@U[1],TypeInfo(TTestCustomJSONGitHubs))<>nil);
!!  U := DynArraySaveJSON(git,TypeInfo(TTestCustomJSONGitHubs));
You can see that the {\f1\fs20 record} serialization is auto-magically available at dynamic array level, which is pretty convenient in our case, since the {\f1\fs20 api.github.com} RESTful service returns a JSON array.
It will convert 160 KB of very verbose JSON information:
$[{"id":8079771,"name":"Component_ZendAuthentication","full_name":"zendframework/Component_ZendAuthentication","owner":{"login":"zendframework","id":296074,"avatar_url":"https://1.gravatar.com/avatar/460576a0866d93fdacb597da4b90f233?d=https%3A%2F%2Fidenticons.github.com%2F292b7433472e2946c926bdca195cec8c.png&r=x","gravatar_id":"460576a0866d93fdacb597da4b90f233","url":"https://api.github.com/users/zendframework","HTTP_url":"https://github.com/zendframework","followers_url":"https://api.github.com/users/zendframework/followers","following_url":"https://api.github.com/users/zendframework/following{/other_user}","gists_url":"https://api.github.com/users/zendframework/gists{/gist_id}","starred_url":"https://api.github.com/users/zendframework/starred{/owner}{/repo}",...
Into the much smaller (6 KB) and readable JSON content, containing only the information we need:
$[
$ {
$  "name": "Component_ZendAuthentication",
$  "id": 8079771,
$  "description": "Authentication component from Zend Framework 2",
$  "fork": true,
$  "owner":
$  {
$   "login": "zendframework",
$   "id": 296074
$  }
$ },
$ {
$  "name": "Component_ZendBarcode",
$  "id": 8079808,
$  "description": "Barcode component from Zend Framework 2",
$  "fork": true,
$  "owner":
$  {
$   "login": "zendframework",
$   "id": 296074
$  }
$ },
$...
During the parsing process, all unneeded JSON members will just be ignored. The parser will jump the data, without doing any temporary memory allocation. This is a huge difference with other existing {\i Delphi} JSON parsers, which first create a tree of all JSON values into memory, then allow to browse all the branches on request.
Note also that the fields have been ordered following the {\f1\fs20 TTestCustomJSONGitHub} record definition, which may not match the original JSON layout (here {\f1\fs20 name/id} fields order is inverted, and {\f1\fs20 owner} is set at the end of each item, for instance).
With {\i mORMot}, you can then access directly the content from your {\i Delphi} code as such:
!  if git[0].id=8079771 then begin
!    Check(git[0].name='Component_ZendAuthentication');
!    Check(git[0].description='Authentication component from Zend Framework 2');
!    Check(git[0].fork=true);
!    Check(git[0].owner.login='zendframework');
!    Check(git[0].owner.id=296074);
!  end;
Note that we do not need to use intermediate objects (e.g. via some obfuscated expressions like {\f1\fs20 gitarray.Value[0].Value['owner'].Value['login']}). Your code will be much more readable, will complain at compilation if you misspell any field name, and will be easy to debug within the IDE (since the {\f1\fs20 record} layout can be easily inspected).
The serialization is able to handle any kind of nested {\f1\fs20 record} or {\i dynamic array}s, including {\i dynamic array}s of simple types (e.g. {\f1\fs20 array of integer} or {\f1\fs20 array of RawUTF8}), or {\i dynamic array}s of {\f1\fs20 record}:
!type
!  TTestCustomJSONRecord = packed record
!    A,B,C: integer;
!    D: RawUTF8;
!    E: record E1,E2: double; end;
!    F: TDateTime;
!  end;
!  TTestCustomJSONArray = packed record
!    A,B,C: integer;
!    D: RawByteString;
!    E: array of record E1: double; E2: string; end;
!    F: TDateTime;
!  end;
!  TTestCustomJSONArraySimple = packed record
!    A,B: Int64;
!    C: array of SynUnicode;
!    D: RawUTF8;
!  end;
The corresponding text definitions may be:
!const
!  __TTestCustomJSONRecord = 'A,B,C integer D RawUTF8 E{E1,E2 double} F TDateTime';
!  __TTestCustomJSONArray  = 'A,B,C integer D RawByteString E[E1 double E2 string] F TDateTime';
!  __TTestCustomJSONArraySimple = 'A,B Int64 C array of synunicode D RawUTF8';
The following types are handled by this feature:
|%30%70
|\b Delphi type|Remarks\b0
|{\f1\fs20 boolean}|Serialized as JSON boolean
|{\f1\fs20 byte word integer cardinal Int64 single @*double@ @*currency@ @*TUnixTime@}|Serialized as JSON number
|{\f1\fs20 string @*RawUTF8@ SynUnicode @*WideString@}|Serialized as JSON string
|{\f1\fs20 DateTime @*TTimeLog@}|Serialized as JSON text, encoded as @*ISO 8601@
|{\f1\fs20 RawByteString}|Serialized as JSON {\f1\fs20 null} or @*Base64@-encoded JSON string
|{\f1\fs20 RawJSON}|Stored as un-serialized raw JSON content\line (e.g. any value, object or array)
|{\f1\fs20 TGUID}|@*GUID@ serialized as JSON text
|nested {\f1\fs20 record}|Serialized as JSON object\line Identified as {\f1\fs20 record ... end;} or  \{ ... \} with its nested definition
|nested {\f1\fs20 registered} record|Serialized as JSON corresponding the the defined callbacks
|{\i dynamic array} of {\f1\fs20 record}|Serialized as JSON array\line Identified as {\f1\fs20 array of ...} or {\f1\fs20 [ ... ]}
|{\i dynamic array} of simple types|Serialized as JSON array\line Identified e.g. as {\f1\fs20 array of integer}
|{\i static array}|Serialized as JSON array\line Handled with enhanced RTTI, not via text definition yet
|{\f1\fs20 variant}|Serialized as JSON, with full support of @80@
|%
For other types (like enumerations or sets), you can simply use the unsigned integer types corresponding to the binary value, e.g. {\f1\fs20 byte word cardinal Int64} (depending on the {\f1\fs20 sizeof()} of the initial value).
For instance, void {\f1\fs20 TTestCustomJSONRecord} may be serialized as:
$ {"A":0,"B":0,"C":0,"D":"","E":{"E1":0,"E2":0},"F":""}
Or void {\f1\fs20 TTestCustomJSONArray} may be serialized as:
$ {"A":0,"B":0,"C":0,"D":null,"E":[],"F":""}
Or void {\f1\fs20 TTestCustomJSONArraySimple} may be serialized as:
$ {"A":0,"B":0,"C":[],"D":""}
You can refer to the supplied regression tests (in {\f1\fs20 TTestLowLevelTypes.EncodeDecodeJSON}) for some more examples of custom JSON serialization.
:53  Dynamic array serialization
:   Standard JSON arrays
Note that @*dynamic array@s are handled in two separated contexts:
- Within the @*ORM@ part of the framework, they are stored as BLOB and always transmitted after {\i Base64} encoding - see @26@;
- Within the scope of {\f1\fs20 interface}-based services, dynamic arrays values and parameters are using the advanced JSON serialization made available in the {\f1\fs20 @*TDynArray@} wrapper, i.e. could be either a true JSON array, or, in default, use generic binary and {\i Base64} encoding, prior to {\i Delphi} 2010.
In fact, this {\f1\fs20 @*TDynArray@} wrapper - see @48@ - recognizes most common kind of {\i dynamic arrays}, like {\f1\fs20 array of byte, word, integer, cardinal, Int64, double, @*currency@, @*RawUTF8@, @*SynUnicode@, WinAnsiString, string}. They will be serialized as a valid JSON array, i.e. a list of valid JSON elements of the matching type (number, floating-point value or string).\line If you have any ideas of standard {\i dynamic arrays} which should be handled, feel free to post your proposal in the forum!
Since {\i Delphi} 2010, the framework will use the @*enhanced RTTI@ to create a JSON array corresponding to the data layout of each {\i dynamic array} item, just as for @51@.
For version of the compiler up to {\i Delphi} 2009, not-known {\i dynamic arrays} (like any {\f1\fs20 array of packed record}) will be serialized by default as binary, then {\i Base64} encoded. This method will always work, but won't be easy to deal with from an AJAX client.
Of course, your applications can supply a custom JSON serialization for any other dynamic array, via the {\f1\fs20 TTextWriter.@**RegisterCustomJSONSerializer@()} class method. Two callbacks are to be defined in association with dynamic array type information, in order to handle proper serialization and un-serialization of the JSON array.\line As an alternative, you can call the {\f1\fs20 @*RegisterCustomJSONSerializerFromText@} method to define the record layout in a convenient text-based format - see above.
In fact, if you register a {\i dynamic array} custom serializer, it will also be used for the associated internal {\f1\fs20 record}.
:   Customized serialization
As we already stated, it may be handy to change the default serialization.
For instance, we would like to serialize a dynamic array of the following record:
!  TFV = packed record
!    Major, Minor, Release, Build: integer;
!    Main, Detailed: string;
!  end;
!  TFVs = array of TFV;
With the default serialization, such a dynamic array will be serialized either:
- As a {\i Base64} encoded binary buffer, before {\i Delphi} 2010 - this won't be easy to understand from an AJAX client, for instance;
- As a JSON array of JSON object, with all property names listed within each object, since {\i Delphi} 2010 and its enhanced RTTI.
This default serialization can be overriden, by defining callbacks. It could be handy, e.g. if you do not like the fact that all field names are written in the data, which may be a waste of space:
# {"Major":1,"Minor":2001,"Release":3001,"Build":4001,"Main":"1","Detailed":"1001"}
In order to add a custom serialization for this kind of record, we need to implement the two needed callbacks.\line Our expected format will be a JSON array of all fields, i.e.:
! [1,2001,3001,4001,"1","1001"]
This layout is more than two times shorter than the default JSON object format.
We may have used another layout, e.g. using {\f1\fs20 JSONEncode()} function and a JSON object layout, or any other valid JSON content.
Here comes the writer:
!class procedure TCollTstDynArray.FVWriter(const aWriter: TTextWriter; const aValue);
!var V: TFV absolute aValue;
!begin
!  aWriter.Add('[%,%,%,%,"%","%"]',
!    [V.Major,V.Minor,V.Release,V.Build,V.Main,V.Detailed],twJSONEscape);
!end;
This event will write one entry of the dynamic array, without the last ',' (which will be appended by {\f1\fs20 TTextWriter. AddDynArrayJSON}). In this method, {\f1\fs20 twJSONEscape} is used to escape the supplied {\f1\fs20 string} content as a valid JSON string (with double quotes and proper @*UTF-8@ encoding).
Of course, the {\i Writer} is easier to code than the {\i Reader} itself:
!class function TCollTstDynArray.FVReader(P: PUTF8Char; var aValue;
!  out aValid: Boolean): PUTF8Char;
!var V: TFV absolute aValue;
!begin // '[1,2001,3001,4001,"1","1001"],[2,2002,3002,4002,"2","1002"],...'
!  aValid := false;
!  result := nil;
!  if (P=nil) or (P^<>'[') then
!    exit;
!  inc(P);
!  V.Major := GetNextItemCardinal(P);
!  V.Minor := GetNextItemCardinal(P);
!  V.Release := GetNextItemCardinal(P);
!  V.Build := GetNextItemCardinal(P);
!  V.Main := UTF8ToString(GetJSONField(P,P));
!  V.Detailed := UTF8ToString(GetJSONField(P,P));
!  if P=nil then
!    exit;
!  aValid := true;
!  result := P; // ',' or ']' for last item of array
!end;
The reader method shall return a pointer to the next separator of the JSON input buffer just after this item (either {\f1\fs20 ','} or {\f1\fs20 ']'}).
The registration process itself is as simple as:
!  TTextWriter.RegisterCustomJSONSerializer(TypeInfo(TFVs),
!    TCollTstDynArray.FVReader,TCollTstDynArray.FVWriter);
Then, from the user code point of view, this dynamic array handling won't change: once registered, the JSON serializers are used everywhere in the framework, as soon as this type is globally registered.
Here is a {\i Writer} method using a JSON object layout, which may be used for {\i Delphi} up to 2009, to obtain a serialization similar to the one generated via the enhanced RTTI.
!class procedure TCollTstDynArray.FVWriter2(const aWriter: TTextWriter; const aValue);
!var V: TFV absolute aValue;
!begin
!  aWriter.AddJSONEscape(['Major',V.Major,'Minor',V.Minor,'Release',V.Release,
!    'Build',V.Build,'Main',V.Main,'Detailed',V.Detailed]);
!end;
This will create some JSON content as such:
# {"Major":1,"Minor":2001,"Release":3001,"Build":4001,"Main":"1","Detailed":"1001"}
We may also use similar callbacks, e.g. if we want the property names to be changed, or ignored depending on some default values.
Then the corresponding {\i Reader} callback could be written as:
!class function TCollTstDynArray.FVReader2(P: PUTF8Char; var aValue;
!  out aValid: Boolean): PUTF8Char;
!var V: TFV absolute aValue;
!    Values: array[0..5] of TValuePUTF8Char;
!begin
!  aValid := false;
!  result := JSONDecode(P,['Major','Minor','Release','Build','Main','Detailed'],@Values);
!  if result=nil then
!    exit; // result^ = ',' or ']' for last item of array
!  V.Major := Values[0].ToInteger;
!  V.Minor := Values[1].ToInteger;
!  V.Release := Values[2].ToInteger;
!  V.Build := Values[3].ToInteger;
!  V.Main := Values[4].ToString;
!  V.Detailed := Values[5].ToString;
!  aValid := true;
!end;
Most of the JSON decoding process is performed within the {\f1\fs20 JSONDecode()} function, which will let {\f1\fs20 Values[].Value/ValueLen} couples point to null-terminated un-escaped content within the {\f1\fs20 P^} buffer. In fact, unserialization will do no memory allocation, and will therefore be very fast.
If you want to go back to the default binary + {\i Base64} encoding serialization, you may run the registering method as such:
!  TTextWriter.RegisterCustomJSONSerializer(TypeInfo(TFVs),nil,nil);
Or calling the text-based registration with a void definition:
! TTextWriter.RegisterCustomJSONSerializerFromText(TypeInfo(TTestCustomJSONGitHub),'');
You can define now your custom JSON serializers, starting for the above code as reference, or via the {\f1\fs20 RegisterCustomJSONSerializerFromText()} method text-based definition.
Note that if the {\i record} corresponding to its item dynamic array has some associated RTTI (i.e. if it contains some reference-counted types, like any {\f1\fs20 string}), it will be serialized as JSON during the {\i mORMot} service process, just as stated with @51@.
:  TSQLRecord TPersistent TStrings TRawUTF8List
Classes with {\f1\fs20 published} properties, i.e. every class inheriting from {\f1\fs20 @*TPersistent@} or our ORM-dedicated {\f1\fs20 @*TSQLRecord@} class will be serialized as a true JSON object, containing all their {\f1\fs20 published} properties values. See @26@ for a corresponding table with the ORM database types and the JSON content.
List of {\i Delphi} strings, i.e. {\f1\fs20 @*TStrings@} kind of classes will be serialized as a JSON array of strings. This is the reason why we also introduced a dedicated {\f1\fs20 @**TRawUTF8List@} class, for direct @*UTF-8@ content storage, via our dedicated {\f1\fs20 RawUTF8} type, reducing the need of encoding conversion, therefore increasing process speed.
:52  TObject serialization
In fact, any {\f1\fs20 @*TObject@} can be serialized as @*JSON@ in the whole framework: not only for the ORM part (for {\f1\fs20 published} properties), but also for SOA (as parameters of interface-based service methods). All JSON @**serialization@ is centralized in {\f1\fs20 ObjectToJSON()} and {\f1\fs20 JSONToObject()} (aka {\f1\fs20 TJSONSerializer.WriteObject}) functions.
:   Custom class serialization
In some cases, it may be handy to have a custom serialization, for instance if you want to manage some third-party classes, or to adapt the serialization scheme to a particular purpose, at runtime.
You can add a customized serialization of any {\f1\fs20 class}, by calling the {\f1\fs20 TJSONSerializer. @**RegisterCustomSerializer@} class method. Two callbacks are to be defined for a specific class type, and will be used to serialize or un-serialize the object instance. The callbacks are class methods ({\f1\fs20 procedure() of object}), and not plain functions (for some evolved objects, it may have sense to use a context during serialization).
In the current implementation of this feature, callbacks expect low-level implementation. That is, their implementation code shall follow function {\f1\fs20 JSONToObject()} patterns, i.e. calling low-level {\f1\fs20 GetJSONField()} function to decode the JSON content, and follow function {\f1\fs20 TJSONSerializer.WriteObject()} patterns, i.e. {\f1\fs20 aSerializer.Add/AddInstanceName/AddJSONEscapeString} to encode the class instance as JSON.
Note that the process is called outside the "{\f1\fs20 \{...\}}" JSON object layout, allowing any serialization scheme: even a class content can be serialized as a JSON string, JSON array or JSON number, on request.
For instance, we'd like to customize the serialization of this class (defined in {\f1\fs20 SynCommons.pas}):
!  TFileVersion = class
!  protected
!    fDetailed: string;
!    fBuildDateTime: TDateTime;
!  public
!    Major: Integer;
!    Minor: Integer;
!    Release: Integer;
!    Build: Integer;
!    BuildYear: integer;
!    Main: string;
!  published
!    property Detailed: string read fDetailed write fDetailed;
!    property BuildDateTime: TDateTime read fBuildDateTime write fBuildDateTime;
!  end;
By default, since it has been defined within {\f1\fs20 \{$M+\} ... \{$M-\}} conditionals, RTTI is available for the {\f1\fs20 published} properties (just as if it were inheriting from {\f1\fs20 TPersistent}). That is, the default JSON serialization will be for instance:
& {"Detailed":"1.2.3.4","BuildDateTime":"1911-03-14T00:00:00"}
This is what is expected when serialized within a {\f1\fs20 TSynLog} content - see @16@ - or for current AJAX use.
We would like to serialize this {\f1\fs20 class} as such:
& {"Major":1,"Minor":2001,"Release":3001,"Build":4001,"Main":"1","BuildDateTime":"1911-03-14"}
We will therefore define the {\i Writer} callback, as such:
!class procedure TCollTstDynArray.FVClassWriter(const aSerializer: TJSONSerializer;
!  aValue: TObject; aOptions: TTextWriterWriteObjectOptions);
!var V: TFileVersion absolute aValue;
!begin
!  aSerializer.AddJSONEscape(['Major',V.Major,'Minor',V.Minor,'Release',V.Release,
!    'Build',V.Build,'Main',V.Main,'BuildDateTime',DateTimeToIso8601Text(V.BuildDateTime)]);
!end;
Most of the JSON serialization work will be made within the {\f1\fs20 AddJSONEscape} method, expecting the JSON object description as an array of name/value pairs.
Then the associated {\i Reader} callback could be, for instance:
!class function TCollTstDynArray.FVClassReader(const aValue: TObject; aFrom: PUTF8Char;
!  var aValid: Boolean; aOptions: TJSONToObjectOptions): PUTF8Char;
!var V: TFileVersion absolute aValue;
!    Values: array[0..5] of TValuePUTF8Char;
!begin
!  result := JSONDecode(aFrom,['Major','Minor','Release','Build','Main','BuildDateTime'],@Values);
!  aValid := (result<>nil);
!  if aValid then begin
!    V.Major := Values[0].ToInteger;
!    V.Minor := Values[1].ToInteger;
!    V.Release := Values[2].ToInteger;
!    V.Build := Values[3].ToInteger;
!    V.Main := Values[4].ToString;
!    V.BuildDateTime := Iso8601ToDateTimePUTF8Char(Values[5].Value,Values[5].ValueLen);
!  end;
!end;
Here, the {\f1\fs20 JSONDecode} function will un-serialize the JSON object into an array of {\f1\fs20 PUTF8Char} values, without any memory allocation (in fact, {\f1\fs20 Values[].Value} will point to un-escaped and #0 terminated content within the {\f1\fs20 aFrom} memory buffer. So decoding is very fast.
Then, the registration step will be defined as such:
!  TJSONSerializer.RegisterCustomSerializer(TFileVersion,
!    TCollTstDynArray.FVClassReader,TCollTstDynArray.FVClassWriter);
If you want to disable the custom serialization, you may call the same method as such:
!  TJSONSerializer.RegisterCustomSerializer(TFileVersion,nil,nil);
This will reset the JSON serialization of the specified class to the default serializer (i.e. writing of {\f1\fs20 published} properties).
The above code uses some low-level functions of the framework (i.e. {\f1\fs20 AddJSONEscape} and {\f1\fs20 JSONDecode}) to implement serialization as a JSON object, but you may use any other serialization scheme, on need. That is, you may serialize the whole class instance just as one JSON string or numerical value, or even a JSON array. It will depend of the implementation of the {\i Reader} and {\i Writer} registered callbacks.
:   Custom field names serialization
If your customization just expect changing some property names, you may use {\f1\fs20 TJSONSerializer.RegisterCustomSerializerFieldNames} class method.
For instance, given the following class:
!type
!  TMyClass = class(TSynPersistent)
!  private
!    FLength: Integer;
!    FColor: Integer;
!    FName: RawUTF8;
!  published
!    property Color: Integer read FColor write FColor;
!    property Length: Integer read FLength write FLength;
!    property Name: RawUTF8 read FName write FName;
!  end;
You may use default serialization as such:
!var
!  O: TMyClass;
!  json: RawUTF8;
!begin
!  O := TMyClass.Create;
!  O.Color := 10;
!  O.Length := 20;
!  O.Name := 'one';
!  json := ObjectToJSON(O);
!  writeln(json); // {"Color":10,"Length":20,"Name":"one"}
Then switch to customized serialization:
!  TJSONSerializer.RegisterCustomSerializerFieldNames(TMyClass, ['name','length'], ['n','len']);
!  json := ObjectToJSON(O);
!  writeln(json); // {"Color":10,"len":20,"n":"one"}
And back to normal/default serialization:
!  TJSONSerializer.RegisterCustomSerializerFieldNames(TMyClass, [], []);
!  json := ObjectToJSON(O);
!  writeln(json); // {"Color":10,"Length":20,"Name":"one"}
You could ignore some fields, by setting the destination name to {\f1\fs20 ''}:
!  TJSONSerializer.RegisterCustomSerializerFieldNames(TMyClass, ['length'], ['']);
!  json := ObjectToJSON(O);
!  writeln(json); // {"Color":10,"Name":"one"}
!  O.Free;
!end;
This method may therefore help working with pre-existing JSON objects, for instance retrieved from a third-party @*REST@ server.
Note that the {\f1\fs20 TJSONSerializer.RegisterCustomSerializerFieldNames} method won't accept {\f1\fs20 TSQLRecord} classes, since {\f1\fs20 ORM} serialization is handled in its own (optimized) set - and you could use ORM-level mapping if needed - see @120@.
:71   TObjectList serialization
You can even serialize {\f1\fs20 @**TObjectList@} instances as a valid JSON array, with the ability to store each instance class name, so allowing the storage of non uniformous lists of objects.\line Calling {\f1\fs20 TJSONSerializer.RegisterClassForJSON()} is just needed to register each {\f1\fs20 TObject} class in its internal tables, and be able to create instances from a {\f1\fs20 class} name serialized in each JSON object.
In fact, if {\f1\fs20 ObjectToJSON()} or {\f1\fs20 TJSONWriter.WriteObject()} have their {\f1\fs20 woStoreClassName} option defined, a new {\f1\fs20 "ClassName":} field will be written as first field of the serialized JSON object.
This new {\f1\fs20 "ClassName"} field will be recognized:
- by {\f1\fs20 JSONToObject()} for {\f1\fs20 TObjectList} members,
- and by the new {\f1\fs20 JSONToNewObject()} method.
Note that all {\f1\fs20 TSQLRecord} classes of a model are automatically registered via a call to {\f1\fs20 TJSONSerializer.RegisterClassForJSON()}: you do not have to register them, and can directly serialize {\f1\fs20 TObjectList} of {\f1\fs20 TSQLRecord}s.
As a consequence, this kind of code can now work:
!// register the type (but Classes.RegisterClass list is also checked)
!TJSONSerializer.RegisterClassForJSON([TComplexNumber]);
!// create an instance by reading the textual class name field
!J := '{"ClassName":"TComplexNumber", "Real": 10.3, "Imaginary": 7.92 }';
!P := @J[1]; // make local copy of constant
!Comp := TComplexNumber(JSONToNewObject(P,Valid));
!// here Comp is a valid unserialized object :)
!Check(Valid);
!Check(Comp.ClassType=TComplexNumber);
!CheckSame(Comp.Real,10.3);
!CheckSame(Comp.Imaginary,7.92);
!// do not forget to free the memory (Comp can be nill if JSON was not valid)
!Comp.Free;
Internal {\f1\fs20 TObjectList} process will therefore rely on a similar process, creating the proper class instances on the fly. You can even have several classes appearing in one {\f1\fs20 TObjectList}: the only prerequisite is that all class types shall have been previously registered on both sides, by a call to {\f1\fs20 TJSONSerializer.RegisterClassForJSON()}.
\page
:9 REST
:  What is REST?
{\i Representational state transfer} (@**REST@) is a style of software architecture for distributed hypermedia systems such as the World Wide Web. As such, it is not just a method for building "web @*service@s". The terms "representational state transfer" and "REST" were introduced in 2000 in the doctoral dissertation of Roy Fielding, one of the principal authors of the Hypertext Transfer Protocol (@**HTTP@) specification, on which the whole Internet rely.
There are 5 basic fundamentals of web which are leveraged to create REST services:
- Everything is a Resource;
- Every Resource is Identified by a Unique Identifier;
- Use Simple and Uniform Interfaces;
- Communication is Done by Representation;
- Every Request is Stateless.
:   Resource-based
Internet is all about getting data. This data can be in a format of web page, image, video, file, etc. It can also be a dynamic output like get customers who are newly subscribed. The first important point in REST is start thinking in terms of resources rather than physical files.
You access the resources via some URI, e.g.
- {\f1\fs20 http://www.mysite.com/pictures/logo.png} - Image Resource;
- {\f1\fs20 http://www.mysite.com/index.html} - Static Resource;
- {\f1\fs20 http://www.mysite.com/Customer/1001} - Dynamic Resource returning XML or JSON content;
- {\f1\fs20 http://www.mysite.com/Customer/1001/Picture} - Dynamic Resource returning an image.
:   Unique Identifier
Older web techniques, e.g. {\i aspx} or {\i ColdFusion}, did request a resource by specifying parameters, e.g.
$ http://www.mysite.com/Default.aspx?a=1;a=2&b=1&a=3
In REST, we add one more constraint to the current URI: in fact, every URI should uniquely represent every item of the data collection.
For instance, you can see the below unique URI format for customer and orders fetched:
|%44%57
|\b Customer data|URI\b0
|Get Customer details with name "dupont"|{\f1\fs20 http://www.mysite.com/Customer/dupont}
|Get Customer details with name "smith"|{\f1\fs20 http://www.mysite.com/Customer/smith}
|Get orders placed by customer "dupont"|{\f1\fs20 http://www.mysite.com/Customer/dupont/Orders}
|Get orders placed by customer "smith"|{\f1\fs20 http://www.mysite.com/Customer/smith/Orders}
|%
Here, "dupont" and "smith" are used as unique identifiers to specify a customer. In practice, a name is far from unique, therefor most systems use an unique ID (like an integer, a hexadecimal number or a @*GUID@).
:   Interfaces
To access those identified resources, basic CRUD activity is identified by a set of HTTP verbs:
|%18%65
|\b HTTP method|Action\b0
|GET|List the members of the collection (one or several)
|PUT|Update a member of the collection
|POST|Create a new entry in the collection
|DELETE|Delete a member of the collection
|%
Then, at URI level, you can define the type of collection, e.g. {\f1\fs20 http://www.mysite.com/Customer} to identify the customers or {\f1\fs20 http://www.mysite.com/Customer/1234/Orders} to access a given order.
This combinaison of HTTP method and URI replace a list of English-based methods, like {\f1\fs20 GetCustomer / InsertCustomer / UpdateOrder / RemoveOrder}.
:   By Representation
What you are sending over the wire is in fact a representation of the actual resource data.
The main representation schemes are XML and @*JSON@.
For instance, here is how a customer data is retrieved from a {\f1\fs20 GET} method:
$$ <Customer>
$$   <ID>1234</ID>
$$   <Name>Dupond</Name>
$$   <Address>Tree street</Address>
$$</Customer>
$$
Below is a simple JSON snippet for creating a new customer record with name and address (since we create a new record, here we named him "Dupond" - with an ending D - not "Dupont"):
$ {"Customer": {"Name":"Dupond", "Address":"Tree street"}}
As a result to this data transmitted with a {\f1\fs20 POST} command, the RESTful server will return the just-created ID.
See @2@ for the reasons why in {\i mORMot}, we prefer to use JSON format.
:   Stateless
Every request should be an independent request so that we can scale up using load balancing techniques.
Independent request means with the data also send the state of the request so that the server can carry forward the same from that level to the next level.
See @15@ for more details.
\page
:  RESTful mORMot
The {\i Synopse mORMot Framework} was designed in accordance with Fielding's REST architectural style without using HTTP and without interacting with the World Wide Web. Such Systems which follow REST principles are often referred to as "RESTful". Optionally, the Framework is able to serve standard HTTP/1.1 pages over the Internet (by using the {\f1\fs20 mORMotHttpClient / mORMotHttpServer} units and the {\f1\fs20 TSQLHttpServer} and {\f1\fs20 TSQLHttpClient} classes), in an embedded low resource and fast HTTP server.
The standard RESTful methods are implemented, i.e. {\f1\fs20 GET/PUT/POST/DELETE}.
The following methods were added to the standard REST definition, for locking individual records and for handling database @*transaction@s (which speed up database process):
- {\f1\fs20 LOCK} to lock a member of the collection;
- {\f1\fs20 UNLOCK} to unlock a member of the collection;
- {\f1\fs20 BEGIN} to initiate a transaction;
- {\f1\fs20 END} to commit a transaction;
- {\f1\fs20 ABORT} to rollback a transaction.
The {\f1\fs20 GET} method has an optional pagination feature, compatible with the YUI DataSource Request Syntax for data pagination - see {\f1\fs20 TSQLRestServer.URI} method and @http://developer.yahoo.com/yui/datatable/#data . Of course, this breaks the "Every Resource is Identified by a Unique Identifier" RESTful principle - but it is much more easy to work with, e.g. to implement paging or custom filtering.
From the {\i Delphi} code point of view, a RESTful @*Client-Server@  architecture is implemented by inheriting some common methods and properties from a main class.
\graph HierTSQLRestClient TSQLRestClient classes hierarchy
\TSQLRestServer\TSQLRest
\TSQLRestClientURI\TSQLRestClient
\TSQLRestClient\TSQLRest
\
This diagram states how the {\f1\fs20 @*TSQLRest@} class implements a common ancestor for both Client and Server classes.
:1   BLOB fields
@**BLOB@ fields are defined as {\f1\fs20 @**TSQLRawBlob@} @*published properties@ in the classes definition - which is an alias to the {\f1\fs20 @*RawByteString@} type (defined in {\f1\fs20 SynCommons.pas} for {\i Delphi} up to 2007, since it appeared only with {\i Delphi} 2009). But their content is not included in standard @*REST@ful methods of the framework, to spare network bandwidth.
The RESTful protocol allows BLOB to be retrieved (GET) or saved (PUT) via a specific URL, like:
$ ModelRoot/TableName/TableID/BlobFieldName
This is even better than the standard @*JSON@ encoding, which works well but convert BLOB to/from hexadecimal values, therefore need twice the normal size of it. By using such dedicated URL, data can be transfered as full binary.
Some dedicated methods of the generic {\f1\fs20 @*TSQLRest@} class handle BLOB fields: {\f1\fs20 RetrieveBlob} and {\f1\fs20 UpdateBlob}.
:   JSON representation
The "{\i 04 - @*HTTP@ @*Client-Server@}" sample application available in the framework source code tree can be used to show how the framework is @*AJAX@-ready, and can be proudly compared to any other @*REST@ server (like {\i CouchDB}) also based on {\f1\fs20 JSON}.
First desactivate the authentication - see @18@ - by changing the parameter from {\f1\fs20 true} to {\f1\fs20 false} in {\f1\fs20 Unit2.pas}:
! DB := TSQLRestServerDB.Create(Model,ChangeFileExt(paramstr(0),'.db3'),
!! false);
and by commenting the following line in {\f1\fs20 Project04Client.dpr}:
!  Form1.Database := TSQLHttpClient.Create(Server,'8080',Form1.Model);
!!  // TSQLHttpClient(Form1.Database).SetUser('User','synopse');
!  Application.Run;
Then you can use your browser to test the JSON content:
- Start the {\f1\fs20 Project04Server.exe} program: the background HTTP server, together with its {\i @*SQLite3@} database engine;
- Start any {\f1\fs20 Project04Client.exe} instances, and add/find any entry, to populate the database a little;
- Close the {\f1\fs20 Project04Client.exe} programs, if you want;
- Open your browser, and type into the address bar:
$  http://localhost:8080/root
- You'll see an error message:
$TSQLHttpServer Server Error 400
- Type into the address bar:
$  http://localhost:8080/root/SampleRecord
- You'll see the result of all {\f1\fs20 SampleRecord} IDs, encoded as a JSON list, e.g.
$ [{"ID":1},{"ID":2},{"ID":3},{"ID":4}]
- Type into the address bar:
$  http://localhost:8080/root/SampleRecord/1
- You'll see the content of the {\f1\fs20 SampleRecord} of ID=1, encoded as JSON, e.g.
${"ID":1,"Time":"2010-02-08T11:07:09","Name":"AB","Question":"To be or not to be"}
- Type into the address bar any other REST command, and the database will reply to your request...
You have got a full HTTP/SQLite3 RESTful JSON server in less than 400 KB.
Note that {\i Internet Explorer} or old versions of {\i FireFox} do not recognize the {\f1\fs20 application/json; charset=UTF-8} content type to be viewed internally. This is a limitation of those softwares, so above requests will download the content as {\f1\fs20 .json} files, but won't prevent AJAX requests to work as expected.
:15   Stateless ORM
Our framework is implementing @*REST@ as a @**stateless@ protocol, just as the @*HTTP@/1.1 protocol it could use as its communication layer.
A {\i stateless} server is a server that treats each request as an independent @*transaction@ that is unrelated to any previous request.
At first, you could find it a bit disappointing from a classic @*Client-Server@ approach. In a stateless world, you are never sure that your Client data is up-to-date. The only place where the data is safe is the server. In the web world, it's not confusing. But if you are coming from a rich Client background, this may concern you: you should have the habit of writing some synchronization code from the server to replicate all changes to all its clients. This is not necessary in a stateless architecture any more.
The main rule of this architecture is to ensure that the Server is the only reference, and that the Client is able to retrieve any pending update from the Server side. That is, always modify a record content on a server side, then refresh the client to retrieve the modified value. Do {\i not} modify the client side directly, but always pass through the Server. The UI components of the framework follow these principles. Client-side modification could be performed, but must be made in a separated autonomous table/database. This will avoid any synchronization problem in case of concurrent client modification.
\page
: REST and JSON
:  JSON format density
Most common @*REST@ful @*JSON@ used a verbose format for the JSON content: see for example @http://bitworking.org/news/restful_json which proposed to put whole URI in the JSON content;
$[
$  "http://example.org/coll/1",
$  "http://example.org/coll/2",
$  "http://example.org/coll/3",
$  ...
$  "http://example.org/coll/N",
$]
The REST implementation of the framework will return most concise JSON content, containing an array of objects:
$ [{"ID":1},{"ID":2},{"ID":3},{"ID":4}]
Depending on a setting, {\i mORMot} servers may in fact returns this alternative (see below {\i non expanded format}), which can be shorter, since it does not replicate field names:
$ {"fieldCount":1,"values":["ID",1,2,3,4,5,6,7]}
which preserves bandwidth and human readability: if you were able to send a GET request to the URI {\f1\fs20 http://example.org/coll} you will be able to append this URI at the beginning of every future request, doesn't it make sense?
In all cases, the {\i Synopse mORMot Framework} always returns the JSON content just as a pure response of a @*SQL@ query, with an array and field names.
:87  JSON (not) expanded layouts
Note that our @*JSON@ content has two layouts, which can be produced according to the {\f1\fs20 TSQLRestServer.NoAJAXJSON} property:
1. the {\i "expanded" or standard/@*AJAX@ layout}, which allows you to create pure {\i @*JavaScript@} objects from the JSON content, because the field name / JavaScript object property name is supplied for every value:
$ [{"ID":0,"Int":0,"Test":"abcde+¬ef+á+¬","Unicode":"abcde+¬ef+á+¬","Ansi":"abcde+¬ef+á+¬","ValFloat":3.14159265300000E+0000,"ValWord":1203,"ValDate":"2009-03-10T21:19:36","Next":0},{..}]
2. the {\i "not expanded" layout}, which reflects exactly the layout of the @*SQL@ request: first line/row are the field names, then all next lines.row are the field content:
$ {"fieldCount":9,"values":["ID","Int","Test","Unicode","Ansi","ValFloat","ValWord","ValDate","Next",0,0,"abcde+¬ef+á+¬","abcde+¬ef+á+¬","abcde+¬ef+á+¬",3.14159265300000E+0000,1203,"2009-03-10T21:19:36",0,..]}
By default, the {\f1\fs20 NoAJAXJSON} property is set to {\f1\fs20 true} when the {\f1\fs20 TSQLRestServer. ExportServerNamedPipe} is called: if you use named pipes for communication, you probably won't use a {\i JavaScript} client since all browsers communicate via @*HTTP@ only!
But otherwise, {\f1\fs20 NoAJAXJSON} property is set to {\f1\fs20 false}. You could force its value to {\f1\fs20 true} and you will save some bandwidth if {\i JavaScript} is never executed: even the parsing of the JSON Content will be faster with {\i Delphi} if JSON content is not expanded.
In this "not expanded" layout, the following JSON content:
$ [{"ID":1},{"ID":2},{"ID":3},{"ID":4},{"ID":5},{"ID":6},{"ID":7}]
will be transfered as shorter:
$ {"fieldCount":1,"values":["ID",1,2,3,4,5,6,7]}
:37  JSON global cache
A global @*cache@, at {\i SQlite3} level, is used to enhance the framework scaling, featuring @*JSON@ storage for its result encoding.
In order to speed-up the server response time, especially in a concurrent client access, the internal database engine is not to be called on every request. In fact, a global cache has been introduced to store in memory the latest @*SQL@ {\f1\fs20 SELECT} statements results, directly in JSON.
The {\i @*SQLite3@} engine access is protected at SQL/JSON cache level, via {\f1\fs20 DB.LockJSON()} calls in most {\f1\fs20 @*TSQLRestServerDB@} methods.
A {\f1\fs20 TSynCache} instance is instantiated within the {\f1\fs20 TSQLDataBase} internal global instance, with the following line:
!constructor TSQLRestServerDB.Create(aModel: TSQLModel; aDB: TSQLDataBase;
!  aHandleUserAuthentication: boolean);
!begin
!  fStatementCache.Init(aDB.DB);
!!  aDB.UseCache := true; // we better use caching in this JSON oriented use
!  (...)
This will enable a global JSON cache at the SQL level. This cache will be reset on every {\f1\fs20 INSERT, UPDATE} or {\f1\fs20 DELETE} SQL statement, whatever the corresponding table is.
If you need to disable the JSON cache for a particular request, add the {\f1\fs20 @*SQLDATABASE_NOCACHE@} text, i.e. the {\f1\fs20 '/*nocache*/'} text comment, anywhere in the SQL statement, e.g. in the ORM WHERE clause. It will indicate to {\f1\fs20 @*TSQLDataBase@} to not cache the returned JSON content. It may be usefull e.g. if you pass a pointer as {\f1\fs20 PtrInt(aVariable)} bound parameter, which may have the very same integer reference value, but diverse content.
In practice, this global cache was found to be efficient, even if its implementation is some kind of "naive". It is in fact much more tuned than other HTTP-level caching mechanisms used in most client-server solutions (using e.g. a {\i Squid} proxy) - since our caching is at the SQL level, it is shared among all @*CRUD@ / @*Rest@ful queries, and is also indenpendent from the authentication scheme, which pollutes the URI. Associated with the other levels of cache - see @38@ - the framework scaling was found to be very good.
:35Client-Server process
%cartoon08.png
:204 Client-Server cheat sheet
Before deeping into the details, and presenting all the {\i mORMot} framework @**Client-Server@ abilities, let's step back, and look at the big picture.
In practice, for your project, you will have several possibilities to create a Client-Server system. ORM, SOA and MVC can all be accessed remotely, and it may not be easy to find out which method is preferred to implement, in the context of a production system.
|%30%30%30
|\b Method|Best for|Beware\b0
|SOA Interfaces|RPC REST|RPC
|SOA Methods|Full REST/HTTP|Verbose
|MVC Web|Web site + AJAX|HTML-oriented
|ORM REST|Tests or internal use|Security/design flaws
|%
In a nutshell,
- {\i SOA Interfaces} - see @63@ - is the preferred way to build both public and private services: both client and server code will be defined from {\f1\fs20 interface} types, including sessions management, stubbing/mocking, documentation generation, and security features.
- {\i SOA Methods} - see @49@ - will open full access to REST/HTTP details of each request, so may be needed to conform to a more REST, less RPC implementation - but the client side will need to be written by hand, and the server side could be more verbose to implement.
- {\i MVC Web} - see @108@ - is the way to go if you expect to develop mostly dynamic web pages, and sometimes consume some {\f1\fs20 JSON} content from {\f1\fs20 JavaScript} if needed, by accessing its {\f1\fs20 url/json} sub path.
- {\i ORM REST} - see @114@ - exposes all data automatically, but should better not be used on production for public APIs for architecture and security reasons, since it is directly tied to the datastore. It could be exposed internally, or for debugging/testing.
- remember that {\i any combination of the four previous framework features} could be defined in the same {\f1\fs20 @*TSQLRestServer@} instance, so you can just pickup what fits best your needs.
We will now present all those communication features, but you may focus on {\i SOA Interfaces}, and its associated samples, when implementing your project, and go back to other details of this exhaustive documentation, only if needed.
: Protocols
The {\i mORMot} framework can be used either @*stand-alone@, or in a @*Client-Server@ model, via several communication layers:
- Fast in-process access (an executable file using a common library, for instance);
- Windows Messages, only locally on the same computer, which are very fast for small content;
- Named pipes, which can be used locally between a Server running as a Windows service and some Client instances;
- @*HTTP@/1.1 over TCP/IP, for remote access.
See @%%mORMotDesign1@ about this Client-Server architecture.
The framework allow you to create either a {\f1\fs20 TSQLHttpClient}, {\f1\fs20 TSQLRestClientURIDll, TSQLRestClientURINamedPipe} or {\f1\fs20 TSQLRestClientURIMessage} instance to access to your data according to the communication protocol used for the server.
Abilities will depend on the protocol used. For instance, HTTP may sounds slower than alternatives, but it is the best protocol for remote access of concurrent clients, even running locally. For instance, {\i mORMot}'s {\f1\fs20 http.sys} based server is able to serve 50,000 concurrent connections without any problem, but you should better not attempt connecting more than a dozen clients via named pipes or messages...
\page
Here are some general information about available communication layers:
|%18%20%20%25%18
| |\b In-process|Windows Messages|Named pipes|HTTP\b0
|Unit|{\f1\fs20 mORMot.pas}|{\f1\fs20 mORMot.pas}|{\f1\fs20 mORMot.pas}|{\f1\fs20 mORMotHttpServer.pas}{\f1\fs20 mORMotHttpClient.pas}
|Speed|****|***|**|*
|Scaling|****|*|*|***
|Hosting|In-process|Local|Local|Remote
|Protocol|Method call|{\f1\fs20 WM_COPYDATA}|\\\\.\\pipe\\mORMot_|Standard
|Data|JSON|JSON|JSON|JSON
|Run as service|Stand alone|No|Yes|Yes
|%
Note that you can have {\i several} protocols exposing the same {\f1\fs20 TSQLRestServer} instance. You may expose the same server over HTTP and over named pipes, at the same time, depending on your speed requirements.
\page
: TSQLRest classes
This architecture is implemented by a hierarchy of classes, implementing the @*REST@ful pattern - see @9@ - for either stand-alone, client or server side, all inheriting from a {\f1\fs20 @*TSQLRest@} common ancestor, as two main branches:
\graph ClientServerRESTClasses RESTful Client-Server classes
\TSQLRestServer\TSQLRest
\TSQLRestClient\TSQLRest
\TSQLRestStorage\TSQLRest
\
All ORM operations (aka @*CRUD@ process) are available from the abstract {\f1\fs20 TSQLRest} class definition, which is overridden to implement either a Server (via {\f1\fs20 TSQLRestServer} classes), or a Client (via {\f1\fs20 TSQLRestClientURI} classes) access to the data.
You should instantiate the classes corresponding to the needed transmission protocol, but should better rely on abstraction, i.e. implement your whole code logic relying on abstract {\f1\fs20 TSQLRestClient / TSQLRestServer} classes. It will then help changing from one protocol or configuration at runtime, depending on your customer's expectations.
:138  Server classes
The following classes are available to implement a {\i Server} instance:
\graph ServerRESTClasses RESTful Server classes
\TSQLRestServerDB\TSQLRestServer
\TSQLRestServerRemoteDB\TSQLRestServer
\TSQLRestServerFullMemory\TSQLRestServer
\TSQLRestServer\TSQLRest
\
In practice, in order to implement the business logic, you should better create a new {\f1\fs20 class}, inheriting from one of the above {\f1\fs20 TSQLRestServer} classes. Having your own inherited class does make sense, especially for implementing your own method-based services - see @49@, or {\f1\fs20 override} internal methods.
The {\f1\fs20 @**TSQLRestServerDB@} class is the main kind of Server of the framework. It will host a {\i @*SQLite3@} engine, as its core @42@.
If your purpose is not to have a full {\i SQLite3} engine available, you may create your server from a {\f1\fs20 @*TSQLRestServerFullMemory@} class instead of {\f1\fs20 TSQLRestServerDB}: this will implement a fast in-memory engine (using {\f1\fs20 TSQLRestStorageInMemory} instances), with basic CRUD features (for ORM), and persistence on disk as JSON or optimized binary files - this kind of server is enough to handle authentication, and host @*service@s in a stand-alone way.
If your services need to have access to a remote ORM server, it may use a {\f1\fs20 @*TSQLRestServerRemoteDB@} class instead: this server will use an internal {\f1\fs20 TSQLRestClient} instance to handle all ORM operations - it can be used e.g. to host some services on a stand-alone server, with all ORM and data access retrieved from another server: it will allow to easily implement a proxy architecture (for instance, as a DMZ for publishing services, but letting ORM process stay out of scope). See @75@ for some hosting scenarios.\line Another option may be to use {\f1\fs20 @*TSQLRestClientRedirect@} - see @186@ - which does something similar, but inheriting from {\f1\fs20 TSQLRestClientURI}.
:  Storage classes
In the {\i mORMot} units, you may also find those classes also inheriting from {\f1\fs20 @*TSQLRestStorage@}:
\graph StorageRESTClasses RESTful storage classes
\TSQLRestStorageExternal\TSQLRestStorage
\TSQLRestStorageMongoDB\TSQLRestStorage
\TSQLRestStorageRecordBased\TSQLRestStorage
\TSQLRestStorageInMemory\TSQLRestStorageRecordBased
\TSQLRestStorageInMemoryExternal\TSQLRestStorageInMemory
\TSQLRestStorageRemote\TSQLRestStorage
\
In the above class hierarchy, the {\f1\fs20 TSQLRestStorage[InMemory][External]} classes are in fact used to store some {\f1\fs20 TSQLRecord} tables in any non-SQL backend:
- {\f1\fs20 TSQLRestStorageExternal} maps tables stored in an external database - see @27@;
- {\f1\fs20 TSQLRestStorageInMemory} stores the data in a {\f1\fs20 TObjectList} - see @57@;
- {\f1\fs20 TSQLRestStorageRemote} will redirect the CRUD operations of a given table to an external {\f1\fs20 TSQLRest} instance (client or server) - see @93@;
- {\f1\fs20 TSQLRestStorageMongoDB} will connect to a remote {\i @*MongoDB@} server to store the tables as a @*NoSQL@ collection of documents - see @83@.
Those classes are used within a main {\f1\fs20 TSQLRestServer} to host some given {\f1\fs20 TSQLRecord} classes, either in-memory, or on external databases. They do not enter in account in our Client-Server presentation, but are implementation details, on the server side.
:  Client classes
A full set of {\i client} classes will implement a RESTful access to a remote database, with associated services and business logic:
\graph ClientRESTClasses RESTful Client classes
rankdir=LR;
\TSQLRestClientURINamedPipe\TSQLRestClientURI
\TSQLRestClientURIMessage\TSQLRestClientURI
\TSQLRestClientURIDll\TSQLRestClientURI
\TSQLRestClientDB\TSQLRestClientURI
\TSQLHttpClientWinINet\TSQLHttpClientWinGeneric
\TSQLHttpClientWinHTTP\TSQLHttpClientWinGeneric
\TSQLHttpClientWinGeneric\TSQLHttpClientGeneric
\TSQLHttpClientWinSock\TSQLHttpClientGeneric
\TSQLHttpClientCurl\TSQLHttpClientWinGeneric
\TSQLHttpClientGeneric\TSQLRestClientURI
\TSQLRestClientRedirect\TSQLRestClientURI
\TSQLRestClientURI\TSQLRestClient
\TSQLRestClient\TSQLRest
\
Of course, all those {\f1\fs20 TSQLRestClient*} classes expect a {\f1\fs20 TSQLRestServer} to be available, via the corresponding transmission protocol.
\page
:186 In-process/stand-alone application
For a @*stand-alone@ application, create a {\f1\fs20 @**TSQLRestClientDB@}. This particular class will initialize an internal {\f1\fs20 @*TSQLRestServerDB@} instance, and you'll have full access to the {\i @*SQLite3@} database in the same process, with no speed penalty.
Content will still be converted to and from JSON, but there will be no delay due to the transmission of the data. Having JSON at hand will enable internal cache - see @39@ - and allow to combine this in-process direct process with other transmission protocols (like named pipes or HTTP).
Another option may be to use {\f1\fs20 @**TSQLRestClientRedirect@}, which allows redirection from any {\f1\fs20 @*TSQLRest@} class, either inheriting from {\f1\fs20 @*TSQLRestClient@} or {\f1\fs20 @*TSQLRestServer@}. Any {\f1\fs20 TSQLRestClientURI.URI} request will be passed to the redirected {\f1\fs20 TSQLRest} instance, which may be local or remote. The {\f1\fs20 TSQLRestClientRedirect.RedirectTo} method allows to enable or disable the redirection at runtime (by setting {\f1\fs20 aRedirected=nil}), or change the redirected {\f1\fs20 TSQLRest} instance on the fly, without creating a new {\f1\fs20 TSQLRestClientRedirect} instance.
You may also directly work with a {\f1\fs20 TSQLRestServerDB} instance, but you will miss some handy features of the {\f1\fs20 TSQLRestClientURI} class, like User-Interface interaction, or advanced ORM/SOA abilities, based on {\f1\fs20 TSQLRestServer.URI} process.
: Local access via named pipes or Windows messages
For a @*Client-Server@ local application, that is some executable running on the same physical machine, create a {\f1\fs20 TSQLRestServerDB} instance, then use the corresponding {\f1\fs20 ExportServer, ExportServerNamedPipe, ExportServerMessage} method to instantiate either a in-process, Named-Pipe or Windows Messages server.
The Windows Messages layer has the lowest overhead and is the fastest transport layer available between several applications on the same computer. But it has the problem of being reserved to desktop applications (since Windows Vista), so you a Windows Messages server won't be accessible when run as a background service.
A named pipe communication is able to be served from a Windows service, and is known to be more efficient when transmitting big messages. So it is the preferred mean of communication for a local application sharing data between clients.
Due to security restriction of newer versions of Windows (i.e. starting with Vista), named pipes are not available by default over a network. This is the reason why this protocol is listed as local access mean only.
\page
:140 Network and Internet access via HTTP
For publishing a server via @*HTTP@/1.1 over TCP/IP, creates a {\f1\fs20 TSQLHttpServer} instance, and associate your running {\f1\fs20 TSQLRestServerDB} to it.
Typical initialization code, as extracted from sample "{\f1\fs20 04 - HTTP Client-Server}", may be:
!  Model := CreateSampleModel;
!  DBServer := TSQLRestServerDB.Create(Model,ChangeFileExt(paramstr(0),'.db3'),true);
!  DBServer.CreateMissingTables;
!!  HttpServer := TSQLHttpServer.Create('8080',[DBServer],'+',HTTP_DEFAULT_MODE);
The following options is usually defined:
!  HttpServer.AccessControlAllowOrigin := '*'; // allow cross-site AJAX queries
And you can optionally define some per domain / per sub-domain hosting redirection:
!  HttpServer.DomainHostRedirect('project.com','root');           // 'root' is current Model.Root
!  HttpServer.DomainHostRedirect('blog.project.com','root/blog'); // MVC application
In all cases, even if HTTP protocol is very network friendly (especially over the 80 port), you shall always acquire IT approval and advices before any deployment over a corporate network, at least to negotiate @*firewall@ settings.
:  HTTP server(s)
The {\f1\fs20 TSQLHttpServer} class is able to use any of two HTTP server classes, as defined in  {\f1\fs20 SynCrtSock} unit - and {\f1\fs20 SynBidirSock} for {\i @*WebSockets@}:
- {\f1\fs20 THttpServer} which is a light and tuned server featuring a thread pool and IOCP implementation pattern, on the raw Sockets API;
- {\f1\fs20 THttpApiServer} which is based on {\i @*http.sys@} API;
- {\f1\fs20 TWebSocketServer} which is a {\f1\fs20 THttpServer} server, able to upgrade to the {\i WebSockets} protocol for asynchronous and bidirectional callbacks - see @150@.
\graph HTTPServers THttpServerGeneric classes hierarchy
\THttpApiServer\THttpServerGeneric
\THttpServer\THttpServerGeneric
\TWebSocketServer\THttpServer
\THttpServerGeneric\TThread
\
On production, {\f1\fs20 THttpApiServer} seems to give the best results, and has a proven and secure implementation. It is also the only one class implementing @*HTTPS@ / @*SSL@ secure communication, if needed. That's why {\f1\fs20 TSQLHttpServer} will first try to use fastest {\i http.sys} kernel-mode server, then fall-back to the generic sockets-based {\f1\fs20 THttpServer} class in case of failure.
You can specify which kind of HTTP server class is to be used, via the {\f1\fs20 aHttpServerKind: TSQLHttpServerOptions} of the {\f1\fs20 TSQLHttpServer.Create} constructor. By default, it will be {\f1\fs20 HTTP_DEFAULT_MODE} (i.e. {\f1\fs20 useHttpApi} over Windows), but you may specify {\f1\fs20 useHttpApiRegisteringURI} for automatic registration of the URI - see @109@ - or {\f1\fs20 useHttpSocket} to use the socket-based {\f1\fs20 THttpServer}, or {\f1\fs20 useBidirSocket} for {\f1\fs20 TWebSocketServer}.
The {\f1\fs20 THttpServerGeneric} abstract class provides one {\f1\fs20 OnRequest} property event, in which all high level process is to take place - it expects some input parameters, then will compute the output content to be sent as response:
!TOnHttpServerRequest = function(Ctxt: THttpServerRequest): cardinal of object;
This event handler prototype is shared by both {\f1\fs20 TThread} classes instances able to implement a {\f1\fs20 HTTP/1.1} server.
Both {\f1\fs20 THttpApiServer} and {\f1\fs20 THttpServer} classes will receive any incoming request, pass it to the {\f1\fs20 TSQLRestServer} instance matching the incoming URI request, via the {\f1\fs20 OnRequest} event handler.
If the request is a remote ORM operation, a @*JSON@ response will be retrieved from the internal @*cache@ of the framework, or computed using the {\i @*SQLite3@} database engine. In case of a remote service access - see @11@ - the request will be computed on the server side, also marshalling the data as JSON. If you specified {\f1\fs20 useBidirSocket} kind of server, you may use remote service access via interfaces, with asynchronous callbacks - see @149@.
The resulting JSON content will be compressed using our very optimized {\i @*SynLZ@} algorithm (20 times faster than Zip/Deflate for compression), if the client is a {\i Delphi} application knowing about {\i SynLZ} - for an @*AJAX@ client, it won't be compressed by default (even if you can enable the deflate algorithm - which may slow down the server).
Then the response will be marked as to be sent back to the Client...
:88  High-performance http.sys server
Since {\i Windows XP SP2} and {\i Windows Server 2003}, the Operating System provides a kernel stack to handle @**HTTP@ requests. This {\f1\fs20 @**http.sys@} driver is in fact a full featured HTTP server, running in kernel mode. It is part of the networking subsystem of the {\i Windows} operating system, as a core component.
The {\f1\fs20 SynCrtSock} unit can implement a HTTP server based on this component. Of course, the {\i Synopse mORMot framework} will use it. If it's not available, it will launch our pure {\i Delphi} optimized HTTP server, using I/O completion ports and a Thread Pool.
What’s good about {\f1\fs20 https.sys}?
- {\i Kernel-mode request queuing}: Requests cause less overhead in context switching, because the kernel forwards requests directly to the correct worker process. If no worker process is available to accept a request, the kernel-mode request queue holds the request until a worker process picks it up.
- {\i Enhanced stability}: When a worker process fails, service is not interrupted; the failure is undetectable by the user because the kernel queues the requests while the WWW service starts a new worker process for that application pool.
- {\i Faster process}: Requests are processed faster because they are routed directly from the kernel to the appropriate user-mode worker process instead of being routed between two user-mode processes, i.e. the good old {\i WinSock} library and the worker process;
- {\i Embedded @*SSL@ process}, when secure @*HTTPS@ communication is needed.
:   Use the http.sys server
Take a look at sample "{\f1\fs20 04 - HTTP Client-Server}", which is able to serve a {\i SQLite3} database content over HTTP, using our RESTful ORM server.\line By default, it will try to use the {\f1\fs20 http.sys} server, then fall-back to plain socket server, in case of failure.
In fact, two steps are performed by the {\f1\fs20 TSQLHttpServer} constructor:
- The HTTP Server API is first initialized (if needed) during {\f1\fs20 THttpApiServer.Create} constructor call. The {\f1\fs20 HttpApi.dll} library (which is the wrapper around {\f1\fs20 http.sys}) is loaded dynamically: so if you are running an old system ({\i Windows XP SP1} for instance), you could still be able to use the server.
- It then tries to register the URI matching the @*REST@ful model - @9@ - via the {\f1\fs20 THttpApiServer.AddUrl} method. In short, the {\f1\fs20 @*TSQLModel@. Root} property is used to compute the RESTful URI needed, just by the book. You can register several {\f1\fs20 TSQLRestServer} instances, each with its own {\f1\fs20 TSQLModel. Root}, if you need it.
As we already stated, if any of those two steps fails (e.g. if {\f1\fs20 http.sys} is not available, or if it was not possible to register the URLs), the {\f1\fs20 TSQLHttpServer} class will fall back into using the other {\f1\fs20 THttpServer} class, which is a plain {\i Delphi} multi-threaded server. It won't be said that we will let you down!
Inside {\f1\fs20 http.sys} all the magic is made... it will listen to any incoming connection request, then handle the headers, then check against any matching URL.
{\f1\fs20 http.sys} will handle all the communication by itself, leaving the server threads free to process the next request.
You can even use a special feature of {\i http.sys} to serve a file content as fast as possible. In fact, if you specify {\f1\fs20 @**HTTP_RESP_STATICFILE@} as {\f1\fs20 Ctxt.OutContentType}, then {\f1\fs20 Ctxt.OutContent} is the @*UTF-8@ file name of a file which must be sent to the client. Note that it will work only with {\f1\fs20 THttpApiServer} kind of server (i.e. using high performance {\i http.sys} API). But whole file access and sending will occur in background, at the kernel level, so with best performance. See sample "{\i 09 - HttpApi web server}" and {\f1\fs20 HttpApiServer.dpr} file.\line If you use a {\f1\fs20 TSQLHttpServer}, the easiest is to define a method-based service - see @49@ - and call {\f1\fs20 Ctxt.ReturnFile()} to return a file content from its name. We will see details about this below. Another possibility may be to override {\f1\fs20 TSQLHttpServer.Request()} method, as stated by {\f1\fs20 Project04ServerStatic.dpr} sample: but we think that a method-based service and {\f1\fs20 Ctxt.ReturnFile()} is preferred.
:109   URI authorization as Administrator
This works fine under XP. Performances are very good, and stability is there. But... here comes the UAC nightmare again.
Security settings have changed since XP. Now only applications running with Administrator rights can register URLs to {\f1\fs20 http.sys}. That is, no real application. So the URI registration step will always fail with the default settings, under Vista and Seven.
The only case when authorization will be possible is when the application launched as a Windows Service, with default services execution user. By default, Windows services are launched with a User which has the Administrator rights.
:    Secure specific authorization
Standard security policy, as requested by Windows for all its {\i http.sys} based systems (i.e. IIS and WCF services) is to explicitly register the URI.
Depending on the system it runs on (i.e. Windows XP or Vista and up), a diverse command line tool is to be used. Can be confusing.
To keep it simple, our {\f1\fs20 SynCrtSock} unit provides a dedicated method to authorize a particular URI prefix to be registered by any user.
Therefore, a program can be easily created and called once with administrator rights to make {\i http.sys} work with our framework. This could be done, for instance, as part of your {\i Setup} program.
Then when your server application will be launched (for instance, as an application in tray icon with normal user rights, or a background Windows service with tuned user rights), it will be able to register all needed URL.
Here is a sample program which can be launched to allow our {\f1\fs20 TestSQL3.dpr} to work as expected - it will allow any connection via the 888 port, using {\f1\fs20 @*TSQLModel@. Root} set as 'root'- that is, an URI prefix of {\f1\fs20 http://+:888/root/} as expected by the kernel server:
!program TestSQL3Register;
!uses
!  SynCrtSock,
!  SysUtils;
!
!// force elevation to Administrator under Vista/Seven
!{$R VistaAdm.res}
!
!begin
!!  THttpApiServer.AddUrlAuthorize('root','888',false,'+'));
!end.
Take also a look at the {\f1\fs20 Project04ServerRegister.dpr} sample, in the context of a whole client/server RESTful solution over HTTP.
Note that you still need to open the IP port for incoming TCP traffic, in the Windows @**firewall@, if you want your server to be accessible to the outer world, as usual.
:    Automatic authorization
An easier possibility could be to run the server application at least once as system Administrator.
The {\f1\fs20 TSQLHttpServer.Create()} constructor has a {\f1\fs20 aHttpServerKind: TSQLHttpServerOptions} parameter. By default, it will be set to {\f1\fs20 @*useHttpApi@}. If you specify {\f1\fs20 useHttpApiRegisteringURI}, the class will register the URI before launching the server process.
All {\i mORMot} samples are compiled with this flag, as such:
! aHTTPServer := TSQLHttpServer.Create(PORT_NAME,[aServer],'+',useHttpApiRegisteringURI);
Note this does not follow default security policy of Windows. But it will make your application development easier.
:    Manual URI authorization
If you configured several {\f1\fs20 http.sys} servers on a given computer, you may have URI registration conflicts after some time.
You can use the {\f1\fs20 netsh} tool to list all registered URL with:
$ netsh http show urlacl
You can optionally specify the fully qualified URL, for instance:
$ netsh http show urlacl url=http://+:80/MyUrl
$ netsh http show urlacl url=http://www.contoso.com:80/MyUrl
Then you can delete any url registration with:
$ netsh http delete urlacl http://host:port/[URI]
The {\f1\fs20 [URI]} is the registered URL path or domain. For instance:
$ netsh delete urlacl url=http://+:80/MyUri
$ netsh delete urlacl url=http://www.contoso.com:80/MyUri
Note that all those commands should be run with administrator user rights.
You can consult the corresponding documentation of {\f1\fs20 netsh http} commands for HTTP context available to query and configure {\f1\fs20 http.sys} settings and parameters at @https://msdn.microsoft.com/en-us/library/windows/desktop/cc307236
:   HTTP API 2.0 Features
Some {\f1\fs20 THttpApiServer} methods are available with the HTTP Server 2.0 API, provided since Windows Vista and Windows Server 2008:
|%40%60
|\b Method|Description\b0
|{\f1\fs20 HasAPI2}|check if the HTTP API 2.0 is available
|{\f1\fs20 SetTimeOutLimits()}|advanced timeout settings
|{\f1\fs20 LogStart()} and {\f1\fs20 LogStop}|HTTP level standard logging
|{\f1\fs20 SetAuthenticationSchemes()}|kernel-mode authentication
|%
Please see the corresponding documentation of {\f1\fs20 SynCrtSock.pas} for further details, and @https://msdn.microsoft.com/en-us/library/windows/desktop/aa364703 as low-level reference of these features. Note that our implementation of {\f1\fs20 http.sys} is more complete than the one currently included in the official .Net WCF framework. Not bad for a third-party library, isn't it?
:135  HTTP client(s)
In fact, there are several implementation of a @**HTTP@/1.1 clients, according to this class hierarchy:
\graph ClientRESTHttpClasses HTTP/1.1 Client RESTful classes
\TSQLHttpClientWinINet\TSQLHttpClientWinGeneric
\TSQLHttpClientWinHTTP\TSQLHttpClientWinGeneric
\TSQLHttpClientWinGeneric\TSQLHttpClientGeneric
\TSQLHttpClientWinSock\TSQLHttpClientGeneric
\TSQLHttpClientCurl\TSQLHttpClientGeneric
\TSQLHttpClientWebsockets\TSQLHttpClientWinSock
\
So you can select either {\f1\fs20 TSQLHttpClientWinSock}, {\f1\fs20 TSQLHttpClientWinINet} or {\f1\fs20 TSQLHttpClientWinHTTP} for a HTTP/1.1 client, under {\i Windows}. By design, {\f1\fs20 TSQLHttpClientWinINet} or {\f1\fs20 TSQLHttpClientWinHTTP} are not available outside of Windows, but {\f1\fs20 TSQLHttpClientCurl} is a great option under Linux, if the {\f1\fs20 @**libcurl@} library is installed, especially if you want to use HTTPS - it will call {\f1\fs20 SynCurl.pas}.\line The {\f1\fs20 TSQLHttpClientWebsockets} class has the ability to {\i upgrade} the HTTP connection to the {\i @*WebSockets@} protocol, which will be used for dual ways callbacks - see @149@.
Each class has its own architecture, and attaches itself to a Windows communication library, all based on {\i WinSock} API. As stated by their name, {\f1\fs20 TSQLHttpClientWinSock} will call directly the {\i WinSock} API, {\f1\fs20 TSQLHttpClientWinINet} will call {\i WinINet} API (as used by IE 6) and {\f1\fs20 TSQLHttpClientWinHTTP} will cal the latest {\i WinHTTP} API:
- {\i WinSock} is the common user-space API to access the sockets stack of Windows, i.e. IP connection - it's able to handle any IP protocol, including TCP/IP, UDP/IP, and any protocol over it (including HTTP);
- {\i WinINet} was designed as an HTTP API client platform that allowed the use of interactive message dialogs such as entering user credentials - it's able to handle HTTP and FTP protocols;
- {\i WinHTTP}'s API set is geared towards a non-interactive environment allowing for use in service-based applications where no user interaction is required or needed, and is also much faster than {\i WinINet} - it only handles HTTP protocol.
\graph ClientHTTPClasses HTTP/1.1 Client architecture
\WinINet\WinSock
\UrlMon\WinINet
\RASAPI\WinINet
\WinHTTP\WinSock
\AutoProxy APIs\WinHTTP
\
Here are some PROs and CONs of the available solutions, under {\i Windows}:
|%25%20%30%25
|\b Criteria|WinSock|WinINet|WinHTTP\b0
|API Level|Low|High|Medium
|Local speed|Fastest|Slow|Fast
|Network speed|Slow|Medium|Fast
|Minimum OS|Win95/98|Win95/98|Win2000
|HTTPS|Not available|Available|Available
|Integration with IE|None|Excellent (proxy)|Available (see below)
|User interactivity|None|Excellent (authentication, dial-up)|None
|%
As stated above, there is still a potential performance issue to use the direct {\f1\fs20 TSQLHttpClientWinSock} class over a network. It has been reported on our forum, and root cause was not identified yet.
Therefore, the {\f1\fs20 TSQLHttpClient} class maps by default to the {\f1\fs20 TSQLHttpClientWinHTTP} class. This is the recommended usage from a {\i Delphi} client application.
Note that even if {\i WinHTTP} does not share by default any @*proxy@ settings with Internet Explorer, it can import the current IE settings.  The {\i WinHTTP} proxy configuration is set by either {\f1\fs20 proxycfg.exe} on Windows XP and Windows Server 2003 or earlier, or {\f1\fs20 netsh.exe} on Windows Vista and Windows Server 2008 or later; for instance, you can run "{\f1\fs20 proxycfg -u}" or "{\f1\fs20 netsh winhttp import proxy source=ie}" to use the current user's proxy settings for Internet Explorer. Under @*64-bit@ Vista/Seven, to configure applications using the 32 bit {\i WinHttp} settings, call {\f1\fs20 netsh} or {\f1\fs20 proxycfg} bits from {\f1\fs20 %SystemRoot%\\SysWOW64} folder explicitly.
Note that by design, the {\f1\fs20 TSQLHttpClient*} classes, like other {\f1\fs20 TSQLRestClientURI} implementations, were designed to be thread safe, since their {\f1\fs20 URI()} method is protected by a lock. See @25@.
:122  HTTPS server
The {\f1\fs20 http.sys} kernel mode server can be defined to serve @**HTTPS@ secure content, i.e. the @**SSL@ protocol over @*HTTP@.
When the {\f1\fs20 aHttpServerSecurity} parameter is set to {\f1\fs20 secSSL} for the {\f1\fs20 TSQLHttpServer.Create()} constructor, the SSL layer will be enabled within {\f1\fs20 http.sys}. Note that {\f1\fs20 useHttpSocket} kind of server does not offer SSL/TLS @*encryption@ yet.
In order to let the SSL layer work as expected, you need first to create and import a set of certificates.
:   Certificates
You need one {\i certificate} (cert) to act as your root authority, and one to act as the actual certificate to be used for the SSL, which needs to be signed by your root authority. If you don't set up the root authority your single certificate won't be trusted, and you will start to discover this through a series of extremely annoying exceptions, long after the fact. To get a free certificate, i.e. for testing purposes, you may use an online service like @http://www.startssl.com
Depending on the {\i Windows} revision you are using, you can run the {\i Internet Information Services (IIS) Manager}: from the {\i Windows} Start menu, click {\f1\fs20 Administrative Tools > Internet Information Services (IIS) Manager}. See @http://support.microsoft.com/kb/299875
You could also install the needed certificate by using some command lines - this may be handy for fast installation using a {\f1\fs20 .bat} file. Here are the needed steps, as detailed in @http://www.codeproject.com/Articles/24027/SSL-with-Self-hosted-WCF-Service and @http://msdn.microsoft.com/en-us/library/ms733791
The following command (run in a {\i Visual Studio} command prompt) will create your root certificate:
$makecert -sv SignRoot.pvk -cy authority -r signroot.cer -a
$    sha1 -n "CN=Dev Certification Authority" -ss my -sr localmachine
Take a look at the above links to see what each of these arguments mean, it isn't terribly important, but it's nice to know.
The {\f1\fs20 MakeCert} tool is available as part of the Windows SDK, which you can download from @http://go.microsoft.com/fwlink/p/?linkid=84091 if you do not want to download the whole {\i Visual Studio} package. Membership in {\i Administrators}, or equivalent, on the local computer is the minimum required to complete this procedure.
Once this command has been run and succeeded, you need to make this certificate a trusted authority. You do this by using the MMC snap in console. Go to the run window and type "{\f1\fs20 mmc}", hit enter. Then in the window that opens (called the "{\i Microsoft Management Console}", for those who care) perform the following actions:
$File -> Add/Remove Snap-in -> Add… -> Double click Certificates -> Select Computer Account and Click Next -> Finish -> Close -> OK
Then select the {\f1\fs20 Certificates (Local Computer) -> Personal -> Certificates} node.
You should see a certificate called "{\f1\fs20 Dev Certificate Authority}" (or whatever else you decided to call it as parameter in the above command line). Move this certificate from the current node to {\f1\fs20 Certificates (Local Computer) -> Trusted Root Certification Authorities -> Certificates} node, drag and drop works happily.
Now you have NOT the cert you need :)\line You have made yourself able to create trusted certs though, which is nice.\line Now you have to create another cert, which you are actually going to use.
Run {\f1\fs20 makecert} again, but run it as follows...
$makecert -iv SignRoot.pvk -ic signroot.cer -cy end -pe -n
$    CN="localhost" -eku 1.3.6.1.5.5.7.3.1 -ss my -sr
$    localmachine -sky exchange -sp
$"Microsoft RSA SChannel Cryptographic Provider" -sy 12
Note that you are using the first certificate as the author for this latest one. This is important... where I have {\f1\fs20 localhost} you need to put the DNS name of your box. In other words, if you deploy your service such that its endpoint reads {\f1\fs20 http://bob:10010/Service} then the name needs to be {\f1\fs20 bob}. In addition, you are going to need to do this for each host you need to run as (yes, so one for {\f1\fs20 bob} and another one for {\f1\fs20 localhost}).
Get the signature of your cert by double clicking on the cert (Select the {\f1\fs20 Certificates (Local Computer) ' Personal ' Certificates}), opening the details tab, and scrolling down to the "{\i Thumbprint}" option.
Select the thumbprint and copy it. Put it in {\f1\fs20 Notepad} or any other text editor and replace the spaces with nothing. {\i Keep this thumbprint heaxdecimal value safe}, since we will need it soon.
You have your certs set up. Congrats!\line But we are not finished yet.
:   Configure a Port with an SSL certificate
Now you get to use another fun tool, {\f1\fs20 httpcfg} (for XP/2003), or its newer version, named aka {\f1\fs20 netsh http} (for Vista/Seven/Eight).
Firstly run the command below to check that you don't have anything running on a port you want.
$httpcfg query ssl
(under XP)
$netsh http show sslcert
(under Vista/Seven/Eight)
If this is your first time doing this, it should just return a newline. If there is already SSL set up on the exact IP you want to use (or if later on you need to delete any mistakes) you can use the following command, where the IP and the port are displayed as a result from the previous query.
Now we have to bind an SSL certificate to a port number, as such (here below, {\f1\fs20 0000000000003ed9cd0c315bbb6dc1c08da5e6} is the {\i thumbprint} of the certificate, as you copied it into the {\f1\fs20 notepad} in the previous paragraph):
$httpcfg set ssl -i 0.0.0.0:8012 -h 0000000000003ed9cd0c315bbb6dc1c08da5e6
(under XP)
$netsh http add sslcert ipport=0.0.0.0:8000 certhash=0000000000003ed9cd0c315bbb6dc1c08da5e6 appid={00112233-4455-6677-8899-AABBCCDDEEFF}
(under Vista/Seven/Eight)\line Here the {\f1\fs20 appid=} parameter is a @*GUID@ that can be used to identify the owning application.
To delete an SSL certificate from a port number previously registered, you can use one of the following commands:
$httpcfg delete ssl -i 0.0.0.0:8005 -h 0000000000003ed9cd0c315bbb6dc1c08da5e6
$httpcfg delete ssl -i 0.0.0.0:8005
(under XP)
$netsh http delete sslcert ipport=0.0.0.0:8005
(under Vista/Seven/Eight)
Note that this is mandatory to first delete an existing certificate for a given port before replacing it with a new one.
:188  Custom Encodings
:   SynLZ/deflate compression
On the client side, the {\f1\fs20 TSQLHttpClientGeneric.Compression} property is by default set as such:
! MyClient.Compression := [hcSynLZ];
It will enable {\i @*SynLZ@} compression in the HTTP headers:
$ ACCEPT-ENCODING: synlz
Our {\i SynLZ} is efficient, especially on @*JSON@ content, and very fast on the server side. It will therefore use less resources than {\f1\fs20 hcDeflate}, so may be preferred when balancing the resource / concurrent client ratio.
You may include {\f1\fs20 hcDeflate} to the property, if you want to support this zip-derivated compression algorithm, e.g. from browsers or any HTTP library. In terms of CPU resources, {\f1\fs20 hcDeflate} will be more consumming than {\f1\fs20 hcSynLZ}, but will obtain a slightly better compression ratio.
If both {\f1\fs20 [hcSynLZ,hcDeflate]} are defined, {\i mORMot} clients will use {\i SynLZ} compression, while other clients (e.g. browsers which do not know about the {\i SynLZ} encoding), will use the standard {\i deflate} compression.
:151   AES encryption over HTTP
In addition to regular HTTPS flow @**encryption@, which is not easy to setup due to the needed certificates, {\i mORMot} proposes a proprietary encryption scheme. It is based on @*SHA256@ and @**AES@256-CFB algorithms, so is known to be secure. You do not need to setup anything on the server or the client configuration, just run the {\f1\fs20 TSQLHttpClient} and {\f1\fs20 TSQLHttpServer} classes with the corresponding parameters.
Note that this encryption uses a global key for the whole process, which should match on both Server and Client sides. You should better hard-code this public key in your Client and Server {\i Delphi} applications, with some variants depending on each end-user service. You can use {\f1\fs20 CompressShaAesSetKey()} as defined in {\f1\fs20 SynCrypto.pas} to set globally this Encryption Key, and an optional Initialization Vector. You can even customize the AES chaining mode, if the default {\f1\fs20 TAESCFB} mode is not what you expect.
When the {\f1\fs20 aHttpServerSecurity} parameter is set to {\f1\fs20 secSynShaAes} for the {\f1\fs20 TSQLHttpServer.Create()} constructor, this proprietary encryption will be enabled on the server side. For instance:
! MyServer := TSQLHttpServer.Create('888',[DataBase],'+',useHttpApi,32,secSynShaAes);
On the client side, you can just set the {\f1\fs20 TSQLHttpClientGeneric.Compression} property as expected:
! MyClient.Compression := [hcSynShaAes];
Once those parameters have been set, a new proprietary encoding will be defined in the HTTP headers:
$ ACCEPT-ENCODING: synshaaes
Then all HTTP body content will be compressed via our {\i SynLZ} algorithm, and encoded using the very secure AES256-CFB scheme.\line On both client and server side, this encryption will use @*AES-NI@ hardware instructions, if available on the CPU it runs on. It ensures that security is enhanced not at the price of performance and scalability.
Since it is a proprietary algorithm, it will work only for {\i Delphi} clients. When accessing for a plain AJAX client, or a {\i Delphi} application with {\f1\fs20 TSQLHttpClientGeneric.Compression = []}, there won't be any encryption at all, due to way HTTP accepts its encoding. For safety, you should therefore use it in conjunction with per-URI Authentication - see @18@.
:   Prefer WebSockets between mORMot nodes
As we just saw, defining {\f1\fs20 hcSynShaAes} will only be available between {\i mORMot} nodes, if both do support the encoding. There is no insurance that content will be encrypted during transmission, e.g. if the client did not define {\f1\fs20 synshaaes}.
Therefore, for truly safe communication between {\i mORMot} nodes, you may consider our {\i @*WebSockets@} client/server implementation instead - see @150@. It implements a proprietary binary protocol for its communication frames, using also {\i SynLZ} compression and @*AES@256-CFB @*encryption@. And, last but not least, it features real-time callbacks, if needed. This kind of access may in fact be considered as the safest available mean of remote connection to a {\i mORMot} server, from stable {\i mORMot} clients, e.g. in a {\i mORMot} @*Cloud@. Then RESTful (AJAX/mobile) clients, may rely on plain HTTP, with {\f1\fs20 hcDeflate} compression.
\page
:25 Thread-safety
We tried to make {\i mORMot} at the same time fast and safe, and able to scale with the best possible performance on the hardware it runs on. @**Multi-thread@ing is the key to better usage of modern multi-core CPUs, and also client responsiveness.
As a result, on the Server side, our framework was designed to be @**thread-safe@.
On typical production use, the {\i mORMot} HTTP server - see @6@ - will run on its own optimized thread pool, then call the {\f1\fs20 TSQLRestServer.URI} method. This method is therefore expected to be thread-safe, e.g. from the {\f1\fs20 TSQLHttpServer. Request} method. Thanks to the @*REST@ful approach of our framework, this method is the only one which is expected to be thread-safe, since it is the single entry point of the whole server. This @*KISS@ design ensure better test coverage.
On the Client side, all {\f1\fs20 TSQLRestClientURI} classes are protected by a global mutex ({\i @**Critical Section@s}), so are thread-safe. As a result, a single {\f1\fs20 TSQLHttpClient} instance can be shared among several threads, even if you may also use one client per thread, as is done with sample 21 - see below, for better responsiveness.
:  Thread safe design
We will now focus on the server side, which is the main strategic point (and potential bottleneck or point of failure) of any {\i Client-Server} architecture.
In order to achieve this thread-safety without sacrificing performance, the following rules were applied in {\f1\fs20 TSQLRestServer.URI}:
- Most of this method's logic is to process the URI and parameters of the incoming request (in {\f1\fs20 TSQLRestServerURIContext.URIDecode*} methods), so is thread-safe by design (e.g. {\f1\fs20 Model} and {\f1\fs20 RecordProps} access do not change during process);
- At @*REST@ful / @*CRUD@ level, {\f1\fs20 Add/Update/Delete/TransactionBegin/Commit/Rollback} methods are locked by default (with a 2 seconds timeout), and {\f1\fs20 Retrieve*} methods are not;
- {\f1\fs20 TSQLRestStorage} main methods ({\f1\fs20 EngineList, EngineRetrieve, EngineAdd, EngineUpdate, EngineDelete, EngineRetrieveBlob, EngineUpdateBlob}) are thread-safe: e.g. {\f1\fs20 @*TSQLRestStorageInMemory@} uses a per-Table {\i Critical Section};
- {\f1\fs20 TSQLRestServerCallBack} method-based services - i.e. @*published method@s of the inherited {\f1\fs20 TSQLRestServer} class as stated @49@ - must be implemented to be thread-safe by default;
- {\f1\fs20 Interface}-based services - see @63@ - have several execution modes, including thread safe automated options (see {\f1\fs20 TServiceMethodOption}) or manual thread safety expectation, for better scaling - see @72@;
- A protected {\f1\fs20 fSessionCriticalSection} is used to protect shared {\f1\fs20 fSession[]} access between clients;
- The {\i @*SQLite3@} engine access is protected at SQL/JSON @*cache@ level, via {\f1\fs20 DB.LockJSON()} calls in {\f1\fs20 @*TSQLRestServerDB@} methods;
- Remote external tables - see @27@ - use thread-safe connections and statements when accessing the databases via SQL;
- Access to {\f1\fs20 fStats} was not made thread-safe, since this data is indicative only: a {\i mutex} was not used to protect this resource.
We tried to make the internal {\i Critical Sections} as short as possible, or relative to a table only (e.g. for {\f1\fs20 TSQLRestStorageInMemory}).
At {\i SQLite3} engine level, there is some kind of "giant lock", so all {\f1\fs20 TSQLDatabase} requests process will be queued. This induces only a slight performance penalty - see @59@ - since the internal SQL/JSON cache implementation needs such a global lock, and since most of the {\i SQLite3} resource use will consist in disk access, which gains to be queued. It also allows to use the {\i SQLite3} engine in {\f1\fs20 lmExclusive} locking mode if needed - see @60@ - with both benefits of high performance and multi-thread friendliness.
From the Client-side, the REST core of the framework is expected to be Client-safe by design, therefore perfectly thread-safe: it is one benefit of the @*stateless@ architecture.
:  Advanced threading settings
You can use {\f1\fs20 TSQLRestServerURI.AcquireExecutionMode[]} property to refine the server-side threading mode. When {\f1\fs20 amLocked} is set, you can also set the {\f1\fs20 AcquireExecutionLockedTimeOut[]} property to specify a wait time to acquire the lock.
The default threading behavior is the following:
|%25%50%25
|\b Command|Description|Default\b0
|{\f1\fs20 execSOAByMethod}|for method-based services|{\f1\fs20 amUnlocked}
|{\f1\fs20 execSOAByInterface}|for interface-based services|{\f1\fs20 amUnlocked}
|{\f1\fs20 execORMGet}|for ORM reads i.e. {\i Retrieve*} methods|{\f1\fs20 amUnlocked}
|{\f1\fs20 execORMWrite}|for ORM writes i.e. {\i Add Update Delete TransactionBegin Commit Rollback} methods|{\f1\fs20 amLocked} +\line timeout of 2000 ms
|%
On need, you can change those settings, to define a particular execution scheme.\line For instance, some external databases (like @*MS SQL@) expect any transaction to be executed within the same connection, so in the same thread context for {\f1\fs20 SynOleDB.pas}, since it uses a per-thread connection pool. When the server is remotely access via HTTP, the incoming requests will be executed from any thread of the HTTP server thread pool. As a result, you won't be able to manage a transaction over MS SQL from the client-side with the default settings.\line To fix it, you can ensure all ORM write operations will be executed in a dedicated background thread, by setting either:
! aServer.AcquireExecutionMode[execORMWrite] := amBackgroundThread;
! aServer.AcquireWriteMode := amBackgroundThread; // same as previous
The same level of thread-safety can be defined for all kind of commands, even if you should better know what you are doing when changing the default settings, since it may create some {\i giant locks} on the server side, therefore voiding any attempt to performance scaling via multi-threading - which is what {\i mORMot} excels in.
At ORM level, with external databases, your {\i mORMot} server may suffer from broken connection to the remote database. To avoid this, you may use {\f1\fs20 ConnectionTimeOutMinutes} property to specify a maximum period of inactivity after which all connections will be flushed and recreated, to avoid potential broken connections issues.\line In this case, you should ensure that all ORM process is blocked so that clearing the connection pool won't break anything in your multi-threaded server. As such, you may set a blocking mode for both {\f1\fs20 execORMGet} and {\f1\fs20 execORMWrite}, for instance:
! aServer.AcquireExecutionMode[execORMGet] := amBackgroundThread;
! aServer.AcquireExecutionMode[execORMWrite] := amBackgroundThread;
The above commands will create one thread for all read operations ({\f1\fs20 execORMGet}), and another thread for all write operations ({\f1\fs20 execORMWrite}). If you want all database access to take place in a {\i single} thread, for both read and write operations, you could write:
! aServer.AcquireExecutionMode[execORMGet] := amBackgroundORMSharedThread;
! aServer.AcquireExecutionMode[execORMWrite] := amBackgroundORMSharedThread;
For instance, this sounds mandatory when using @*Jet/MSAccess@ as external database, since its implementation seems not thread-safe: if you write in one thread, then read immedially in another thread, the Jet engine is not able to find the just written data from the 2nd thread. This is clearly a bug of the Jet engine - but setting {\f1\fs20 amBackgroundORMSharedThread} option to circumvent the issue.
During any @*ORM@ or @*SOA@ process, you can access the current execution context from the {\f1\fs20 @*ServiceContext@ threadvar} variable, as stated @107@. For instance, you can retrieve the current logged user, or its session ID.
In practice, {\f1\fs20 execSOAByMethod} may benefit of a per-method locking, {\f1\fs20 execSOAByInterface} of using its own execution options - see @72@, and {\f1\fs20 execORMGet} to be let unlocked to allow concurrent reads of all connected clients.
:  Proven behavior
When we are talking about thread-safety, nothing compares to a dedicated stress test program. An average human brain (like ours) is not good enough to ensure proper design of such a complex process. So we have to prove the abilities of our little {\i mORMot}.
In the supplied regression tests, we designed a whole class of multi-thread testing, named {\f1\fs20 TTestMultiThreadProcess}. Its methods will run every and each Client-Server protocols available (direct access via {\f1\fs20 TSQLRestServerDB} or {\f1\fs20 TSQLRestCLientDB}, Windows Messages, named pipes, and both HTTP servers - i.e. {\f1\fs20 http.sys} based or WinSock-based)- see @35@.
Each protocol will execute in parallel a list of INSERTs - i.e. {\f1\fs20 TSQLRest.Add()} - followed by a list of SELECTs - i.e. {\f1\fs20 TSQLRest.Retrieve()}. Those requests will be performed in 1 thread, then 2, 5, 10, 30 and 50 concurrent threads. The very same {\i SQLite3} database (in {\f1\fs20 lmExclusive} locking mode) is accessed at once by all those clients. Then the IDs generated by each thread are compared together, to ensure no cross-insertion did occur during the process.
Those automated tests did already reveal some issues in the initial implementation of the framework. We fixed any encountered problems, as soon as possible. Feel free to send us any feedback, with code to reproduce the issue: but do not forget that multi-threading is also difficult to test - problems may occur not in the framework, but in the testing code itself!
When setting {\f1\fs20 OperationCount} to 1000 instead of the default 200, i.e. running 1000 INSERTions and 1000 SELECTs in concurrent threads, the numbers are the following, on the local machine (compiled with {\i Delphi} XE4):
$ Multi thread process:
$  - Create thread pool: 1 assertion passed  3.11ms
$  - TSQLRestServerDB: 24,061 assertions passed  903.31ms
$     1=41986/s  2=24466/s  5=14041/s  10=9212/s  30=10376/s  50=10028/s
$  - TSQLRestClientDB: 24,062 assertions passed  374.93ms
$     1=38606/s  2=35823/s  5=30083/s  10=32739/s  30=33454/s  50=30905/s
$  - TSQLRestClientURINamedPipe: 12,012 assertions passed  1.68s
$     1=4562/s  2=5002/s  5=3177/s
$  - TSQLRestClientURIMessage: 16,022 assertions passed  616.00ms
$     1=16129/s  2=24873/s  5=8613/s  10=11857/s
$  - TSQLHttpClientWinHTTP_HTTPAPI: 24,056 assertions passed  1.63s
$     1=5352/s  2=7441/s  5=7563/s  10=7903/s  30=8413/s  50=9106/s
$  - TSQLHttpClientWinSock_WinSock: 24,061 assertions passed  1.10s
$     1=11528/s  2=10941/s  5=12014/s  10=12039/s  30=9443/s  50=10831/s
$  Total failed: 0 / 124,275  - Multi thread process PASSED  6.31s
For direct in-process access, {\f1\fs20 TSQLRestClientDB} sounds the best candidate: its abstraction layer is very thin, and much more multi-thread friendly than straight {\f1\fs20 TSQLRestServerDB} calls. It also will feature a cache, on need - see @38@. And it will allow your code to switch between {\f1\fs20 TSQLRestClientURI} kind of classes, from its shared abstract methods.
Named pipes and Windows Messages are a bit constrained in highly parallel mode, but HTTP does pretty good. The server based on {\f1\fs20 http.sys} (HTTP API) is even impressive: the more clients, the more responsive it is. It is known to scale much better than the WinSock-based class supplied, which shines with one unique local client (i.e. in the context of those in-process regression tests), but sounds less reliable on production.
:4  Highly concurrent clients performance
In addition, you can make yourself an idea, and run the "{\i 21 - HTTP Client-Server performance}" sample programs, locally or over a network, to check the {\i mORMot} abilities to scale and serve a lot of clients with as few resources as possible.
Compile both client and server projects, then launch {\f1\fs20 Project21HttpServer.exe}. The server side will execute as a console window.
This Server will define the same {\f1\fs20 TSQLRecordPeople} as used during our multi-thread regression tests, that is:
!type
!  TSQLRecordPeople = class(TSQLRecord)
!  private
!    fFirstName: RawUTF8;
!    fLastName: RawUTF8;
!    fYearOfBirth: integer;
!    fYearOfDeath: word;
!  published
!    property FirstName: RawUTF8 read fFirstName write fFirstName;
!    property LastName: RawUTF8 read fLastName write fLastName;
!    property YearOfBirth: integer read fYearOfBirth write fYearOfBirth;
!    property YearOfDeath: word read fYearOfDeath write fYearOfDeath;
!  end;
The server main block is just the following:
!!  aModel := TSQLModel.Create([TSQLRecordPeople]);
!  try
!    aDatabaseFile := ChangeFileExt(paramstr(0),'.db3');
!    DeleteFile(aDatabaseFile);
!!    aServer := TSQLRestServerDB.Create(aModel,aDatabaseFile);
!    try
!!      aServer.DB.Synchronous := smOff;
!!      aServer.DB.LockingMode := lmExclusive;
!      aServer.NoAJAXJSON := true;
!      aServer.CreateMissingTables;
!      // launch the server
!!      aHTTPServer := TSQLHttpServer.Create('888',[aServer]);
!      try
!        writeln(#13#10'Background server is running at http://localhost:888'#13#10+
!                #13#10'Press [Enter] to close the server.');
!        ConsoleWaitForEnterKey;
!      finally
!        aHTTPServer.Free;
!      end;
!    finally
!      aServer.Free;
!    end;
!  finally
!    aModel.Free;
!  end;
Following the @10@ pattern, {\f1\fs20 aServer} will give remote CRUD access to the {\f1\fs20 TSQLRecordPeople} table (as defined in {\f1\fs20 aModel}), from HTTP. We defined {\f1\fs20 Synchronous := smOff} and {\f1\fs20 LockingMode := lmExclusive} to have the best performance possible, as stated by @60@. Our purpose here is not to have true ACID behavior, but test concurrent remote access.
The Client is just a RAD form which will execute the very same code than during the regression tests, i.e. a {\f1\fs20 TTestMultiThreadProcess class} instance, as shown by the following code:
!    Tests := TSynTestsLogged.Create;
!!    Test := TTestMultiThreadProcess.Create(Tests);
!    try
!!      Test.ClientOnlyServerIP := StringToAnsi7(lbledtServerAddress.Text);
!!      Test.MinThreads := ThreadCount;
!!      Test.MaxThreads := ThreadCount;
!!      Test.OperationCount := OperationCount;
!!      Test.ClientPerThread := ClientPerThread;
!!      Test.CreateThreadPool;
!      txt := Format
!        ('%s'#13#10#13#10'Test started with %d threads, %d client(s) per thread and %d rows to be inserted...',
!        [txt,ThreadCount,ClientPerThread,OperationCount]);
!      mmoInfo.Text := txt;
!      Timer.Start;
!!      Test._TSQLHttpClientWinHTTP_HTTPAPI;
!      txt := mmoInfo.Text+Format(#13#10'Assertion(s) failed: %d / %d'+
!        #13#10'Number of clients connected at once: %d'+
!        #13#10'Time to process: %s'#13#10'Operation per second: %d',
!        [Test.AssertionsFailed,Test.Assertions,
!         ThreadCount*ClientPerThread,Timer.Stop,Timer.PerSec(OperationCount*2)]);
!      mmoInfo.Text := txt;
!    finally
!      Test.Free;
!      Tests.Free;
!    end;
Each thread of the thread pool will create its own HTTP connection, then loop to insert ({\f1\fs20 Add} ORM method) and retrieve ({\f1\fs20 Retrieve} ORM method) a fixed number of objects - checking that the retrieved object fields match the inserted values. Then all generated IDs of all threads are checked for consistency, to ensure no race condition did occur.
The input parameters are therefore the following:
- Remote HTTP server IP (port is 888);
- Number of client threads;
- Number of client instances per thread;
- Number of {\f1\fs20 TSQLRecordPeople} objects added.
When running over the following hardware configuration:
- Server is a Core i7 Notebook, with SSD, under Windows 7;
- Client is a Core 2 Duo Workstation, with regular hard-drive (not used), under Windows 7;
- Communicating over a somewhat slow 100 Mb network with a low priced Ethernet HUB.
Typical results are the following:
|%12%20%20%20%15%15
|{\b Threads|Clients/thread|Rows inserted|Total Clients|Time (sec)|Op/sec}
|1|1|10000|1|15.78|1267
|50|1|10000|50|2.96|6737
|100|1|10000|100|3.09|6462
|100|1|20000|100|6.19|6459
|50|2|100000|100|34.99|5714
|100|2|100000|200|36.56|5469
|500|100|100000|50000|92.92|2152
|%
During all tests, no assertion failed, meaning that no concurrency problem did occur, nor any remote command lost. The {\i @*SQlite3@} core, exposes via the {\i mORMot} server, outputs data at an amazing pace of 6000 op/sec - i.e. comparable to most high-end databases. It is worth noting that when run several times in a row, the same set of input parameters give the very same speed results: it indicates that the architecture is pretty stable and could be considered as safe. The system is even {\i able to serve 50000 connected clients at once}, with no data loss - in this case, performance is lower (2152 insert/second in the above table), but we clearly reached the CPU and network limit of our client hardware configuration; in the meanwhile, server CPU resources on the Notebook server did have still some potential, and RAM consumption was pretty slow.
Average performance is pretty good, even more if we consider that we are inserting one object per request, with no transaction. In fact, it sounds like if our little {\i SQLite3} server is faster than most database servers, even when accessed in highly concurrent mode! In batch mode - see @28@ - we may achieve amazing results.
Feel free to send your own benchmark results and feedback, e.g. with concurrent clients on several workstations, or long-running tests, on our forums.
:114Client-Server ORM
%cartoon01.png
As stated above, all ORM features can be accessible either stand-alone, or remotely via some dedicated @35@.
That is, CRUD operations can be executed either at the database level, or remotely, from the same methods defined in {\f1\fs20 TSQLRest} abstract class.
This feature has several benefits, among them:
- No need to deploy the database client library for your application clients - a standard IP network connection is enough;
- Therefore the client application can safely remain small, and stand-alone - no installation step is necessary, and you still have the full power of a native rich client;
- Clients access their objects in an abstract way, i.e. without any guess on how persistence is handled: some classes may be stored in one {\i @*SQlite3@} database, others may exist only in server's memory, others may be stored e.g. in an external {\i @*Oracle@}, {\i @*Firebird@}, {\i @*PostgreSQL@, @*MySQL@, @*DB2@, @*Informix@} or {\i @*MS SQL@} database;
- You can switch from local to remote access just by changing the class type, even at runtime;
- Optimization is implemented at every level of the @*n-Tier@ architecture, e.g. cache or security.
: ORM as local or remote
Typical Client-Server @*REST@ful POST / Add request over HTTP/1.1 will be implemented as such, on both Client and Server side:
\graph ArchClient Client-Server implementation - Client side
subgraph cluster {
\Client.Add\TSQLHttpClient.URI\ORM to JSON over REST
\TSQLHttpClient.URI\Client.Add\return¤new ID
label = "Client";
}
\TSQLHttpClient.URI\Http Server\HTTP protocol¤POST
\Http Server\TSQLHttpClient.URI
\
\graph ArchServer Client-Server implementation - Server side
\HTTP Client\Http Server\HTTP protocol¤POST
\Http Server\HTTP Client
subgraph cluster {
\Http Server\TSQLRestServer.URI\dispatch
\TSQLRestServer.URI\TSQLRestServerDB.EngineAdd\decode¤POST JSON
\TSQLRestServerDB.EngineAdd\SQLite3 SQL\SQL insert
\SQLite3 SQL\SQLite3 BTREE\prepare + execute
\SQLite3 BTREE\Database file\atomic write
\SQLite3 BTREE\TSQLRestServer.URI\return new ID
\TSQLRestServer.URI\Http Server\return 200 OK + ID
label = "Server";
}
\
Of course, several clients can access to the same server.
The same server is also able to publish its RESTful services over several communication protocol at once, e.g. HTTP/1.1 for remote access over a network (either corporate or the Internet), named pipes or Windows Messages for fast local access.
The above diagram describes a direct INSERT into the Server's main {\i @*SQLite3@} engine, but other database back-ends are available - see @42@.
It is possible to by-pass the whole Client-Server architecture, and let the application be @*stand-alone@, by defining a {\f1\fs20 @*TSQLRestClientDB@} class, which will embed a {\f1\fs20 @*TSQLRestServerDB@} instance in the same executable:
\graph ArchStandAlone Client-Server implementation - Stand-Alone application
subgraph cluster {
\Client.Add\TSQLRestClientDB.URI\ORM to JSON over REST
\TSQLRestClientDB.URI\TSQLRestServerDB.URI\direct call
\TSQLRestServerDB.URI\TSQLRestServerDB.EngineAdd\decode¤POST JSON
\TSQLRestServerDB.EngineAdd\SQLite3 SQL\SQL insert
\SQLite3 SQL\SQLite3 BTREE\prepare + execute
\SQLite3 BTREE\Database file\atomic write
\SQLite3 BTREE\TSQLRestServerDB.URI\return new ID
\TSQLRestServerDB.URI\Client.Add\return new ID
label = "Stand-Alone application";
}
\
In fact, the same executable could be launched as server, as stand-alone application, or even client application! It is just a matter of how you initialize your {\f1\fs20 TSQLRest} classes instances - see @35@. Some {\i mORMot} users use this feature to ease deployment, support and configuration. It can be also extremely useful at debugging time, since you may run the server and client side of your project at once within the same application, from the IDE.
\page
In case of a @*Virtual Table@ use (either in-memory or for accessing an external database), the client side remains identical. Only the server side is modified as was specified by @30@:
\graph ArchServerVirtual Client-Server implementation - Server side with Virtual Tables
subgraph cluster {
\Http Server\TSQLRestServer.URI\dispatch
\TSQLRestServer.URI\TSQLRestServerDB.EngineAdd\decode¤POST JSON
\TSQLRestServerDB.EngineAdd\SQLite3 SQL\SQL insert
\SQLite3 SQL\SQLite3 engine\prepare + execute
\SQLite3 engine\SQLite3 BTREE\Is a SQLite3 table?
\SQLite3 BTREE\Database file\atomic write
\SQLite3 BTREE\TSQLRestServer.URI\return new ID
\SQLite3 engine\TSQLVirtualTableExternal\Is a Virtual table?
\TSQLVirtualTableExternal\TSQLDBConnectionProperties\external database
\TSQLDBConnectionProperties\TSQLDBStatement\compute SQL insert
\TSQLDBStatement\OleDB/ODBC or other\execute SQL
\OleDB/ODBC or other\Database Client\store data
\Database Client\OleDB/ODBC or other
\OleDB/ODBC or other\TSQLDBStatement
\TSQLDBStatement\TSQLVirtualTableExternal
\TSQLVirtualTableExternal\TSQLRestServer.URI\return new ID
\TSQLRestServer.URI\Http Server\return 200 OK + ID
label = "Server";
}
\
In fact, the above function correspond to a database @*model@ with only external virtual tables, and with {\f1\fs20 StaticVirtualTableDirect=false}, i.e. calling the Virtual Table mechanism of {\i SQlite3} for each request.
But most of the time, i.e. for RESTful / @*CRUD@ commands, the execution is more direct:
\graph ArchVirtualDirect Client-Server implementation - Server side with "static" Virtual Tables
subgraph cluster {
\Http Server\TSQLRestServer.URI\dispatch
\TSQLRestServer.URI\TSQLRestServerDB.EngineAdd\Is a SQLite3 table?
\TSQLRestServer.URI\TSQLRestStorageExternal.EngineAdd\Is a static table?
\TSQLRestServerDB.EngineAdd\SQLite3 SQL\SQL insert
\SQLite3 SQL\SQLite3 engine\Is a SQLite3 table?
\SQLite3 engine\SQLite3 BTREE\prepare + execute
\SQLite3 BTREE\Database file\atomic write
\SQLite3 BTREE\TSQLRestServer.URI\return new ID
\TSQLRestStorageExternal.EngineAdd\TSQLDBConnectionProperties\external database
\TSQLDBConnectionProperties\TSQLDBStatement\compute SQL
\TSQLDBStatement\OleDB/ODBC or other\execute SQL
\OleDB/ODBC or other\Database Client\store data
\Database Client\OleDB/ODBC or other
\OleDB/ODBC or other\TSQLDBStatement
\TSQLDBStatement\TSQLRestStorageExternal.EngineAdd
\TSQLRestStorageExternal.EngineAdd\TSQLRestServer.URI\return new ID
\TSQLRestServer.URI\Http Server\return 200 OK + ID
label = "Server";
}
\
As stated in @27@, the @*static@ {\f1\fs20 TSQLRestStorageExternal} instance is called for most RESTful access. In practice, this design will induce no speed penalty, when compared to a direct database access. It could be even faster, if the server is located on the same computer than the database: in this case, use of JSON and REST could be faster - even faster when using @28@.
In order to be exhaustive, here is a more complete diagram, showing how native {\i @*SQLite3@}, in-memory or external tables are handled on the server side. You'll find out how CRUD statements are handled directly for better speed, whereas any SQL @*JOIN@ query can also be processed among all kind of tables.
\graph ArchServerFull Client-Server implementation - Server side
subgraph cluster {
\Http Server\TSQLRestServer.URI\dispatch
\TSQLRestServer.URI\TSQLRestServerDB.Engine*\Refers to¤a SQLite3 table or¤a JOINed query?
\TSQLRestServer.URI\TSQLRestStorage.¤Engine*\CRUD over¤a static table?
\TSQLRestServerDB.Engine*\SQLite3 SQL\decode into SQL
\SQLite3 SQL\SQLite3 engine\prepare + execute
\SQLite3 engine\SQLite3 BTREE\For any¤SQLite3¤table
\SQLite3 engine\TSQLRestServer.URI
\SQLite3 BTREE\Database file\atomic¤read/write
\SQLite3 BTREE\SQLite3 engine
\SQLite3 engine\TSQLVirtualTableJSON¤TSQLVirtualTableBinary\For any in-memory¤Virtual Table
\SQLite3 engine\TSQLVirtualTableExternal\For any external¤Virtual table
\TSQLVirtualTableExternal\TSQLDBConnectionProperties\external¤database
\TSQLDBConnectionProperties\TSQLDBStatement\compute SQL
\TSQLDBStatement\OleDB/ODBC or other\execute SQL
\OleDB/ODBC or other\Database Client\handle data
\Database Client\OleDB/ODBC or other
\OleDB/ODBC or other\TSQLDBStatement
\TSQLDBStatement\SQLite3 engine
\TSQLRestServer.URI\Http Server\return 200 OK +¤result (JSON)
\TSQLRestStorage.¤Engine*\TSQLRestStorageExternal.¤Engine*\Is an external¤Virtual Table?
\TSQLRestStorage.¤Engine*\TSQLRestStorageInMemory.¤Engine*\Is an in-memory table?¤(Virtual or static)
\TSQLRestStorage.¤Engine*\TSQLRestStorageMongoDB.¤Engine*\Is a MongoDB table?
\TSQLRestStorage.¤Engine*\TSQLRestStorageRemote.¤Engine*\Is a redirected table?
\TSQLRestStorageExternal.¤Engine*\TSQLDBConnectionProperties¤TSQLDBStatement\compute SQL
\TSQLRestStorageRemote.¤Engine*\Remote TSQLRestClientHTTP¤Local TSQLRestServerDB
\TSQLDBConnectionProperties¤TSQLDBStatement\OleDB/ODBC¤or other\execute SQL
\OleDB/ODBC¤or other\TSQLRestServer.URI
\OleDB/ODBC¤or other\Database¤Client\handle data
\Database¤Client\OleDB/ODBC¤or other
\TSQLRestStorageInMemory.¤Engine*\CRUD¤in-memory¤TSQLRecord
\CRUD¤in-memory¤TSQLRecord\TSQLRestServer.URI
\TSQLVirtualTableJSON¤TSQLVirtualTableBinary\Process¤in-memory¤TSQLRecord
\Process¤in-memory¤TSQLRecord\SQLite3 engine
\TSQLRestStorageMongoDB.¤Engine*\TMongoCollection¤MongoDB Client
\TMongoCollection¤MongoDB Client\TSQLRestServer.URI
label = "Server";
}
\
You will find out some speed numbers resulting from this unique architecture in the supplied @59@.
\page
: Stateless design
:  Server side synchronization
Even if @15@, it's always necessary to have some event triggered on the server side when a record is edited.
On the server side, you can use this method prototype:
!type
!  ///  used to define how to trigger Events on record update
!  // - see TSQLRestServer.OnUpdateEvent property
!  // - returns true on success, false if an error occured (but action must continue)
!  TNotifySQLEvent = function(Sender: TSQLRestServer; Event: TSQLEvent;
!    aTable: TSQLRecordClass; aID: TID): boolean of object;
!
!  TSQLRestServer = class(TSQLRest)
! (...)
!    /// a method can be specified here to trigger events after any table update
!    OnUpdateEvent: TNotifySQLEvent;
:  Client side synchronization
But if you want all clients to be notified from any update, there is no direct way of broadcasting some event from the server to all clients.
It's not even technically possible with pipe-oriented transport layer, like named pipes or the TCP/IP - @*HTTP@ protocol.
What you can do easily, and is what should be used in such case, is to have a timer in your client applications which will call {\f1\fs20 TSQLRestClientURI. UpdateFromServer()} method to refresh the content of any {\f1\fs20 @*TSQLRecord@} or {\f1\fs20 @*TSQLTableJSON@} instance:
!/// check if the data may have changed of the server for this objects, and
!// update it if possible
!// - only working types are TSQLTableJSON and TSQLRecord descendants
!// - make use of the InternalState function to check the data content revision
!// - return true if Data is updated successfully, or false on any error
!// during data retrieval from server (e.g. if the TSQLRecord has been deleted)
!// - if Data contains only one TSQLTableJSON, PCurrentRow can point to the
!// current selected row of this table, in order to refresh its value
!function UpdateFromServer(const Data: array of TObject; out Refreshed: boolean;
!  PCurrentRow: PInteger = nil): boolean;
With a per-second timer, it's quick and reactive, even over a remote network.
The @*stateless@ aspect of @*REST@ allows this approach to be safe, by design.
This is handled natively by our Client User Interface classes, with the following parameter defining the User interface:
!/// defines the settings for a Tab
!  TSQLRibbonTabParameters = object
!  (...)
!    /// by default, the screens are not refreshed automaticaly
!    // - but you can enable the auto-refresh feature by setting this
!    // property to TRUE, and creating a WM_TIMER timer to the form
!    AutoRefresh: boolean;
This parameter will work only if you handle the {\f1\fs20 WM_TIMER} message in your main application form, and call {\f1\fs20 Ribbon.WMRefreshTimer}.
See for example this method in the main demo (@!Lib\SQLite3\Samples\MainDemo\FileMain.pas@ unit):
!procedure TMainForm.WMRefreshTimer(var Msg: TWMTimer);
!begin
!  Ribbon.WMRefreshTimer(Msg);
!end;
In a multi-threaded client application, and even on the server side, a @*stateless@ approach makes writing software easier. You do not have to care about forcing data refresh in your client screens. It's up to the screens to get refreshed. In practice, I found it very convenient to rely on a timer instead of calling the somewhat "delicate" {\f1\fs20 TThread. Synchronize} method.
:  Let applications be responsive
All the client communication is executed by default in the current thread, i.e. the main thread for a typical GUI application.
Since all communication is performed in {\i blocking} mode, if the remote request takes long to process (due to a bad/slow network, or a long server-side action), the application may become unresponsive, from the end-user experience. Even {\i Windows} may be complaining about a "non responsive application", and may propose to kill the process, which is far away from an expected behavior.
In order to properly interacts with the user, a {\f1\fs20 OnIdle} property has been defined in {\f1\fs20 TSQLRestClientURI}, and will change the way communication is handled. If a callback event is defined, all client communication will be processed in a background thread, and the current thread (probably the main UI thread) will wait for the request to be performed in the background, running the {\f1\fs20 OnIdle} callback in loop in the while.
You can find in the {\f1\fs20 mORMotUILogin} unit two methods matching this callback signature:
!  TLoginForm = class(TForm)
!  (...)
!    class procedure OnIdleProcess(Sender: TSynBackgroundThreadAbstract; ElapsedMS: Integer);
!    class procedure OnIdleProcessForm(Sender: TSynBackgroundThreadAbstract; ElapsedMS: Integer);
!  end;
The first {\f1\fs20 OnIdleProcess()} callback will change the mouse cursor shape to {\f1\fs20 crHourClass} after a defined period of time. The {\f1\fs20 OnIdleProcessForm()} callback won't only change the mouse cursor, but also display a pop-up window with a {\i 'Please wait...'} message, if the request takes even more time. Both will call {\f1\fs20 Application.ProcessMessages} to ensure the application User Interface is still responsive.
Some global variable were also defined to tune the behavior of those two callbacks:
!var
!  /// define when TLoginForm.OnIdleProcess() has to display the crHourGlass cursor
!  // after a given time elapsed, in milliseconds
!  // - default is 100 ms
!  OnIdleProcessCursorChangeTimeout: integer = 100;
!
!  /// define when TLoginForm.OnIdleProcessForm() has to display the temporary
!  // form after a given time elapsed, in milliseconds
!  // - default is 2000 ms, i.e. 2 seconds
!  OnIdleProcessTemporaryFormTimeout: integer = 2000;
!
!  /// define the message text displayed by TLoginForm.OnIdleProcessForm()
!  // - default is sOnIdleProcessFormMessage resourcestring, i.e. 'Please wait...'
!  OnIdleProcessTemporaryFormMessage: string;
You can therefore change those settings to customize the user experience. We tested it with a 3 second artificial temporizer for each request, and the applications were running smoothly, even if slowly - but comparable to most @*Web Application@s, in fact. The {\i SynFile} main demo (available in the {\f1\fs20 SQlite3\\Samples\\MainDemo} folder) defines such a callback.
Note that this {\f1\fs20 OnIdle} feature is defined at {\f1\fs20 TSQLRestClientURI} class level, so is available for all communication protocols, not only HTTP but named pipes or in-process, so could be used to enhance user experience in case of some time consuming process.
\page
:28 BATCH sequences for adding/updating/deleting records
:  BATCH process
When use the so-called @**BATCH@ sequences?
In a standard @*Client-Server@ architecture, especially with the common understanding (and most implementations) of a @*REST@ful service, any {\f1\fs20 Add / Update / Delete} method call requires a back and forth flow to then from the remote server. A so-called {\i round-trip} occurs: a message is sent to the client, the a response is sent back to the client.
In case of a remote connection via the Internet (or a slow network), you could have up to 100 ms of latency: it's just the "ping" timing, i.e. the time spent for your IP packet to go to the server, then back to you.
If you are making a number of such calls (e.g. add 1000 records), you'll have 100*1000 ms = 100 s = 1:40 min just because of this network latency!
\graph BATCHRoundTrip1 BATCH mode Client-Server latency
\ORM CRUD operation\ORM HTTP Client\In-process¤no latency
\ORM HTTP Client\ORM CRUD operation
\ORM HTTP Client\ORM HTTP Server\Internet¤100 ms latency
\ORM HTTP Server\ORM HTTP Client
\ORM HTTP Server\ORM database core\In-process¤no latency
\ORM database core\ORM HTTP Server
\
The BATCH sequence allows you to regroup those statements into just ONE remote call. Internally, it builds a @*JSON@ stream, then post this stream at once to the server. Then the server answers at once, after having performed all the modifications.
Some new {\f1\fs20 TSQLRestClientURI} methods have been added to implement BATCH sequences to speed up database modifications: after a call to {\f1\fs20 BatchStart}, database modification statements are added to the sequence via {\f1\fs20 BatchAdd / BatchUpdate / BatchDelete}, then all statements are sent as one to the remote server via {\f1\fs20 BatchSend} - this is MUCH faster than individual calls to {\f1\fs20 Add / Update / Delete} in case of a slow remote connection (typically @*HTTP@ over Internet).
Since the statements are performed at once, you can't receive the result (e.g. the ID of the added row) on the same time as you append the request to the BATCH sequence. So you'll have to wait for the {\f1\fs20 BatchSend} method to retrieve all results, {\i at once}, in a {\i dynamic} {\f1\fs20 array of TID}.
As you may guess, it's also a good idea to use a @*transaction@ for the whole process. By default, the BATCH sequence is not embedded into a transaction.
You have two possibilities to add a transaction:
- Either let the caller use an explicit {\f1\fs20 TransactionBegin} ... {\f1\fs20 try}... {\f1\fs20 Commit  except RollBack} block;
- Or specify a number of rows as {\f1\fs20 @*AutomaticTransactionPerRow@} parameter to {\f1\fs20 BatchStart()}: in this case, a transaction will be emitted (up to the specified number of rows) on the server side. You can just set {\f1\fs20 maxInt} if you want all rows to be modified in a single transaction.
This second method is preferred, since defining transactions from the client side is not a good idea: it may block other clients attempts to create their own transaction.
Here is typical use (extracted from the regression @*test@s in {\f1\fs20 SynSelfTests.pas}:
!  // start the BATCH sequence
!!  Check(ClientDist.BatchStart(TSQLRecordPeople,1000));
!  // now a transaction will be created by chunk of 1000 modifications
!  // delete some elements
!  for i := 0 to n-1 do
!!    Check(ClientDist.BatchDelete(IntArray[i])=i);
!  // update some elements
!  nupd := 0;
!  for i := 0 to aStatic.Count-1 do
!  if i and 7<>0 then
!  begin // not yet deleted in BATCH mode
!    Check(ClientDist.Retrieve(aStatic.ID[i],V));
!    V.YearOfBirth := 1800+nupd;
!!    Check(ClientDist.BatchUpdate(V)=nupd+n);
!    inc(nupd);
!  end;
!  // add some elements
!  V.LastName := 'New';
!  for i := 0 to 1000 do
!  begin
!    V.FirstName := RandomUTF8(10);
!    V.YearOfBirth := i+1000;
!!    Check(ClientDist.BatchAdd(V,true)=n+nupd+i);
!  end;
!  // send the BATCH sequences to the server
!!  Check(ClientDist.BatchSend(Results)=200);
!  // now all data has been commited on the server
!  // now Results[] contains the results of every BATCH statement...
!  Check(Length(Results)=n+nupd+1001);
!  // Results[0] to Results[n-1] should be 200 = deletion OK
!  // Results[n] to Results[n+nupd-1] should be 200 = update OK
!  // Results[n+nupd] to Results[high(Results)] are the IDs of each added record
!  for i := 0 to high(Results) do
!    if i<nupd+n then
!      Check(Results[i]=200) else
!    begin
!      Check(Results[i]>0);
!      ndx := aStatic.IDToIndex(Results[i]);
!      Check(ndx>=0);
!      with TSQLRecordPeople(aStatic.Items[ndx]) do
!      begin
!        Check(LastName='New','BatchAdd');
!        Check(YearOfBirth=1000+i-nupd-n);
!      end;
!    end;
!  // check ClientDist.BatchDelete(IntArray[i]) did erase the record
!  for i := 0 to n-1 do
!    Check(not ClientDist.Retrieve(IntArray[i],V),'BatchDelete');
In the above code, all @*CRUD@ operations are performed as usual, using {\f1\fs20 BatchAdd BatchDelete BatchUpdate} methods instead of plain {\f1\fs20 Add Delete Update} methods. The ORM will take care of all the low-level data process, including JSON serialization, automatic per-chunk transactions creation, and SQL statements generation, with several optimizations - see @78@ and @99@.
In the above example, we started the batch process involving only {\f1\fs20 TSQLRecordPeople} kind of objects:
!  Check(ClientDist.BatchStart(TSQLRecordPeople,1000));
But you could mix any kind of {\f1\fs20 TSQLRecord} content, if you set the class to {\f1\fs20 nil}, as such:
!  Check(ClientDist.BatchStart(nil,1000));
or use the {\f1\fs20 BatchStartAny()} method:
!  Check(ClientDist.BatchStartAny(1000));
In practice, you should better create and maintain your own instance of {\f1\fs20 TSQLRestBatch}, so that you will be able to implement any number of simultaneous batch process - see @100@.
:  Transmitted JSON
As described above, all {\f1\fs20 Batch*()} methods do serialize the objects values as JSON on the client side, then send this JSON at once to the server, where it will be processed without any client-server {\i round-trip} and slow latency.
Here is some extract of typical JSON stream as sent to the server:
${"People":["DELETE",2,"DELETE",13,"DELETE",24,
$   (...)  all DELETE actions
$  ,"DELETE",11010,
$  "PUT",{"RowID":3,"FirstName":"Sergei1","LastName":"Rachmaninoff","YearOfBirth":1800, "YearOfDeath":1943},
$  "PUT",{"RowID":4,"FirstName":"Alexandre1","LastName":"Dumas","YearOfBirth":1801, "YearOfDeath":1870},
$   (...)  all PUT = update actions
$  "PUT",{"RowID":11012,"FirstName":"Leonard","LastName":"da VinÃ§i","YearOfBirth":9025, "YearOfDeath":1519},
$  "POST",{"FirstName":"â€š@â€¢Å"Hâ€ mÂ£Â g","LastName":"New","YearOfBirth":1000, "YearOfDeath":1519},
$  "POST",{"FirstName":"@â€¦,KAÂ½Ã #Â¶f","LastName":"New","YearOfBirth":1001, "YearOfDeath":1519},
$   (...)  all POST = add actions
$ "POST",{"FirstName":"+ÂtqCXW3Ã‚\"","LastName":"New","YearOfBirth":2000, "YearOfDeath":1519}
$  ]}
If {\f1\fs20 BatchAdd} implies only simple fields (which is the default), those fields name won't be transmitted, and the following will be emitted in the JSON stream, to reduce needed bandwith:
$  "SIMPLE",["â€š@â€¢Å"Hâ€ mÂ£Â g","New",1000,1519],
By default, @*BLOB@ fields are excluded from the {\f1\fs20 Batch} content: only simple fields are send. But {\f1\fs20 BatchAdd BatchUpdate} methods (or corresponding {\f1\fs20 TSQLRestBatch} {\f1\fs20 Add} or {\f1\fs20 Update} methods) could contain a custom list of fields to be transmitted, in which you could specify any {\f1\fs20 TSQLRawBlob} field: the binary BLOB content will be encoded as {\i @*Base64@} within the JSON process, and you may definitively gain some resource and speed in such case. Of course, all the data should be small enough to be stored in memory, so the BLOB fields should better be up to some dozen of MB - use several {\f1\fs20 Batch} instances in a loop, if you have a huge set of data.
On success, the following JSON stream will be received from the server:
$ [200,200,...]
This array of results is either the HTTP status codes (here 200 means OK), or the inserted new ID (for a {\f1\fs20 BatchAdd} command).
All the JSON generation (client-side) and parsing (server-side) has been optimized to minimize the resource needed. With the new internal {\i @*SynLZ@} compression (available by default in our @*HTTP@ Client-Server classes), used bandwidth is minimal.
Thanks to this BATCH process, most time is now spent into the database engine itself, and not in the communication layer.
:100  Unit Of Work pattern
:   Several Batches
On the {\f1\fs20 TSQLRestClientURI} side, all {\i BatchStart/BatchAdd/BatchUpdate/BatchDelete} methods are using a single temporary storage during the BATCH preparation. This may be safe only if one single thread is accessing the methods - which is usually the case for a @*REST@ Client application.
In fact, all BATCH process is using a {\f1\fs20 @*TSQLRestBatch@} class, which can be created on the fly, and safely coexist as multiple instances for the same {\f1\fs20 TSQLRest}. As a result, you can create your own local {\f1\fs20 TSQLRestBatch} instances, for safe batch process. This is in fact mandatory on the {\f1\fs20 TSQLRestServer} side, which do not have the {\f1\fs20 Batch*()} methods, since they will not be thread safe.
On the server side, you may write for instance:
!var Batch: TSQLRestBatch;
!    IDs: TIntegerDynArray;
!...
!  Batch := TSQLRestBatch.Create(Server,TSQLRecordTest,30);
!  try
!    for i := 10000 to 10099 do begin
!      R.Int := i;
!      R.Test := Int32ToUTF8(i);
!      Check(Batch.Add(R,true)=i-10000);
!    end;
!    Check(Server.BatchSend(Batch,IDs)=HTTP_SUCCESS);
!  finally
!    Batch.Free;
!  end;
The ability to handle several {\f1\fs20 TSQLRestBatch} classes in the same time will allow to implement the {\i @**Unit Of Work@} pattern. It can be used to maintain a list of objects affected by a business transaction and coordinates the writing out of changes and the resolution of concurrency problems, especially in a complex @*SOA@ application with a huge number of connected clients.
In a way, you can think of the {\i Unit of Work} as a place to dump all transaction-handling code.\line The responsibilities of the {\i Unit of Work} are to:
- Manage transactions;
- Order the database inserts, deletes, and updates;
- Prevent concurrency problems;
- Group requests to maximize the database performance.
The value of using a {\i Unit of Work} pattern is to free the rest of your code from these concerns so that you can otherwise concentrate on business logic.
:   Updating only the mapped fields
In practice, the {\f1\fs20 BatchUpdate} method will only update the mapped fields if called on a record in which a {\f1\fs20 FillPrepare} was performed, and not unmapped (i.e. with no call to {\f1\fs20 FillClose}). This is required for coherency of the retrieval/modification process.
For instance, in the following code, {\f1\fs20 V.FillPrepare} will retrieve only {\f1\fs20 ID} and {\f1\fs20 YearOfBirth} fields of the {\f1\fs20 TSQLRecordPeople} table, so subsequent {\f1\fs20 BatchUpdate(V)} calls will only update the {\f1\fs20 YearOfBirth} field:
!  // test BATCH update from partial FillPrepare
!!  V.FillPrepare(ClientDist,'LastName=:("New"):','ID,YearOfBirth');
!  if ClientDist.TransactionBegin(TSQLRecordPeople) then
!  try
!    Check(ClientDist.BatchStart(TSQLRecordPeople));
!    n := 0;
!    V.LastName := 'NotTransmitted';
!    while V.FillOne do begin
!      Check(V.LastName='NotTransmitted');
!      Check(V.YearOfBirth=n+1000);
!      V.YearOfBirth := n;
!!      ClientDist.BatchUpdate(V); // will update only V.YearOfBirth
!      inc(n);
!    end;
!  (...)
The transmitted JSON will be computed as such on the client side:
$  ....,"PUT",{"RowID":324,"YearOfBirth":1000},...
And the generated SQL on the server side will be:
$ UPDATE People SET YearOfBirth=? WHERE RowID=?
$ ... with bound parameters: [1000,324]
As a result, BATCH process could be seen as a good way of implementing {\i @*Unit Of Work@} for your business layer - see @102@.\line You will be able to modify all your objects as requested, with high-level OOP methods, then have all data transmitted and processed at once when {\f1\fs20 BatchSend()} is called. The {\f1\fs20 BatchStart} - {\f1\fs20 BatchSend} - {\f1\fs20 BatchAbort} commands will induce a safe transactional model, relying on the client side for tracking the object modifications, and optimizing the database process on the server side as a simple "save and forget" task, to any SQL or @*NoSQL@ engine.
Note that if several {\f1\fs20 ClientDist.BatchUpdate(V)} commands are executed within the same {\f1\fs20 FillPrepare()} context, they will contain the same fields ({\f1\fs20 RowID} and {\f1\fs20 YearOfBirth}). They will therefore generate the same statement ({\f1\fs20 UPDATE People SET YearOfBirth=? WHERE RowID=?}), which will benefit of {\i Array Binding} on the database side - see @78@ - if available.
Here is some code, extracted from "web blog" sample "{\i 30 - MVC Server}", which will update an integer array mapped into a table. All {\f1\fs20 TSQLTag.Occurence} integers are stored in a local {\f1\fs20 TSQLTags.Lookup[].Occurence} dynamic array, which will be used to display the occurence count of each tag of the articles.\line The following method will first retrieve ID and Occurence from the database, and update the {\f1\fs20 TSQLTag.Occurence} if the internal dynamic array contains a new value.
!procedure TSQLTags.SaveOccurence(aRest: TSQLRest);
!var tag: TSQLTag;
!    batch: TSQLRestBatch;
!begin
!  Lock.ProtectMethod;
!  TAutoFree.Several([
!    @tag,TSQLTag.CreateAndFillPrepare(aRest,'','RowID,Occurence'),
!    @batch,TSQLRestBatch.Create(aRest,TSQLTag,1000)]);
!  while tag.FillOne do begin
!    if tag.ID<=length(Lookup) then
!      if Lookup[tag.ID-1].Occurence<>tag.Occurence then begin
!        tag.Occurence := Lookup[tag.ID-1].Occurence;
!        batch.Update(tag); // will update only Occurence field
!      end;
!  end;
!  aRest.BatchSend(batch);
!end;
In the above code, you can identify:
- {\f1\fs20 CreateAndFillPrepare} + {\f1\fs20 FillOne} methods are able to retrieve all values of the {\f1\fs20 TSQLTag} class, and iterate easily over them;
- A local {\f1\fs20 TSQLRestBatch} is prepared, and will store locally - via {\f1\fs20 batch.Update()} - any modification; as we already stated, only the retrieved field (i.e. {\f1\fs20 'Occurence'}) will be marked as to be updated;
- {\f1\fs20 aRest.BatchSend(batch)} will send all new values (if any) to the server, in a single network round trip, and a single transaction;
- This method is made thread safe by using {\f1\fs20 Lock.ProtectMethod} ({\f1\fs20 Lock} is a mutex private to the {\f1\fs20 TSQLTags} instance);
- Local variables are allocated and automatically relased when the method exits, using {\f1\fs20 TAutoFree.Several()} - see @130@ - which avoid to write two nested {\f1\fs20 try .. finally Free end} loops.
Such a pattern is very common in {\i mORMot}, and illustrate how high-level ORM methods can be used instead of manual SQL. With the potential benefit of a much better performance, and cleaner code.
:  Local network as bottleneck
When using a remote database on a physical network, a {\i round-trip} delay occurs for each request, this time between the ORM server side and the external Database engine.
\graph BATCHRoundTrip2 BATCH mode latency issue on external DB
\ORM CRUD operation\ORM HTTP Client\In-process¤no latency
\ORM HTTP Client\ORM CRUD operation
\ORM HTTP Client\ORM HTTP Server\Internet¤100 ms latency
\ORM HTTP Server\ORM HTTP Client
\ORM HTTP Server\ORM database core\In-process¤no latency
\ORM database core\ORM HTTP Server
\ORM database core\External DB\Local Network¤1 ms latency
\External DB\ORM database core
\
At first, the 1 ms latency due to the external database round-trip may sound negligible. @28@ did already shortcut the Internet latency, which was much higher.
But in a @17@, most of the process is done on the server side: the slightest execution delay will induce a noticeable performance penalty. In practice, you won't be able to achieve more than 500-600 requests per second when performing individual {\f1\fs20 INSERT}, {\f1\fs20 DELETE} or {\f1\fs20 UPDATE} statements over any SQL database. Even if run locally on the same server, most SQL databases will suffer from the overhead of inter-process communications, achieving 6,000-7,000 update requests per second at best.
Your customers may not understand why using a {\i @*SQLite3@} engine will be much faster than a dedicated {\i @*Oracle@} instance they do pay huge amount of money for, since {\i SQLite3} runs locally in the ORM server process. One common solution is to use stored procedures, or tune the SQL for your database - but you will loose most of the ORM and SOA benefits - see @101@.
Of course, {\i mORMot} can do better than that. Its ORM will automatically use two ways of diminishing the number of round-trips to the database:
- Using {\i Array Binding} - see @78@;
- Or {\i multi-INSERT statements} - see @99@.
Both methods will group all the transmitted data in chunks, as much as possible. Performance will therefore increase, reaching 50,000-60,000 writes per second, depending on the database abilities.
Those features are enabled by default, and the fastest method will always be selected by the ORM core, as soon as it is available on the database back-end. You do not have to worry about configuring your application. Just enjoy its speed.
:78   Array binding
:    For faster BATCH mode
When used in conjunction with @27@, BATCH methods can be implemented as {\i @**array bind@ing} if the corresponding {\f1\fs20 TSQLDBConnection} class implements the feature. By now, only {\f1\fs20 SynDBOracle}, {\f1\fs20 SynDBZeos} and {\f1\fs20 SynDBFireDAC} units implement it.
Our {\f1\fs20 SynDB.pas} unit offers some {\f1\fs20 TSQLDBStatement.BindArray()} methods, introducing native {\i array binding} for faster database batch modifications. It is working in conjunction with our the BATCH methods of the ORM, so that CRUD modification actions will transparently be grouped within one {\i round-trip} over the network.
Thanks to this enhancement, inserting records within {\i Oracle} (over a 100 Mb Ethernet network) comes from 400-500 rows per second to more than 70,000 rows per second, according to our @59@.
The great maintainers of the {\i ZEOS Open Source} library did especially tune its internals to support {\i mORMot} at its full speed, directly accessing the {\i ZDBC} layer - see @94@. The {\i ZEOS 7.2} branch did benefit of a huge code refactoring, and also introduced {\i array binding} abilities. This feature will be recognized and handled by our ORM, if available at the ZDBC provider side. Today, only the ZDBC {\i Oracle} and {\i Firebird} providers do support this feature. But the list is growing.
The @*FireDAC@ (formerly @*AnyDAC@) library is the only one implementing this feature (known as {\i Array DML} in the {\i FireDAC} documentation) around all available {\i Delphi} commercial libraries. Enabling it gives a similar performance boost, not only for {\i Oracle}, but also {\i @*MS SQL@, @*Firebird@, @*DB2@, @*MySQL@, @*Informix@} and {\i @*PostgreSQL@}.
In practice, when accessing {\i Oracle}, our own direct implementation in {\f1\fs20 SynDBOracle} still gives better performance results than the {\i ZDBC} / {\i FireDAC} implementation.
In fact, some modern database engine (e.g. {\i Oracle} or MS SQL) are even faster when using {\i array binding}, not only due to the network latency reduce, but to the fact that in such operations, integrity checking and indexes update is performed at the end of the bulk process. If your table has several indexes and constraints, it will make using this feature even faster than a "naive" stored procedure executing individual statements within a loop.
:    For faster IN clause
Sometimes, you want to write {\f1\fs20 SELECT} statements with a huge {\f1\fs20 IN} clause. If the number of the items in the {\f1\fs20 IN} expression is stable, you may benefit for a prepared statement, e.g.
$ SELECT * FROM MyTable WHERE ID IN [?,?,?,?,?]
But if the IDs are not fixed, you should have to create an expression without any parameter, or use a temporary table:
$ SELECT * FROM MyTable WHERE ID IN [1,4,8,12,24,27]
As an alternative, {\f1\fs20 SynDBOracle} provides the ability to bind an array of parameters which may be cast to an {\i Oracle} Object, so that you could use it as a single parameter.\line Current implementation support either {\f1\fs20 TInt64DynArray} or {\f1\fs20 TRawUTF8DynArray} values, as such:
!var
!  arr: TInt64DynArray = [1, 2, 3];
!Query := TSQLDBOracleConnectionProperties.NewThreadSafeStatementPrepared(
!  'select * from table where table.id in'+
!    '(select column_value from table(cast(? as SYS.ODCINUMBERLIST)))');
!Query.BindArray(1, arr);
!Query.ExecutePrepared;
{\f1\fs20 RawUTF8} arrays are also supported (which can be used as fall back in case {\f1\fs20 Int64} arrays are not supported by the client, e.g. with {\i Oracle} 10):
!var
!  arr: TRawUTF8DynArray = ['123123423452345', '3124234454351324', '53567568578867867'];
!Query := TSQLDBOracleConnectionProperties.NewThreadSafeStatementPrepared(
!  'select * from table where table.id in'+
!    '(select column_value from table(cast(? as SYS.ODCIVARCHAR2LIST)))');
!Query.BindArray(1, arr);
!Query.ExecutePrepared;
From tests on production, this implementation is 2-100 times faster (depending on array and table size) and also simpler, compared to temporary table solution.\line Drawback is that it is supported by {\f1\fs20 SynDBOracle} only by now.
:99   Optimized SQL for bulk insert
Sadly, array binding is not available for all databases or libraries.\line In order to maximize speed, during BATCH insertion, the {\i mORMot} ORM kernel is able to generate some optimized SQL statements, depending on the target database, to send several rows of data at once. It induces a noticeable speed increase when saving several objects into an external database.
Automatic multi-INSERT statement generation is available for:
- Our internal {\i SQLite3} engine (in the {\f1\fs20 mORMotSQLite3.pas} unit);
- Almost all the supported @27@ (in the {\f1\fs20 mORMotDB.pas} unit): {\i @*SQlite3@} (3.7.11 and later), {\i @*MySQL@, @*PostgreSQL@, @*MS SQL@ Server} (2008 and up), {\i @*Oracle@, @*Firebird@, @*DB2@, @*Informix@} and {\i @*NexusDB@} - and since it is implemented at SQL level, it is available for all supported access libraries, e.g. {\i @*ODBC@, @*OleDB@, Zeos/@*ZDBC@, @*UniDAC@};
- And, in the {\i @*NoSQL@} form of "documents array" insertion, for the {\i @*MongoDB@} database (in the {\f1\fs20 mORMotMongoDB.pas} unit).
It means that even providers not implementing array binding (like {\i OleDB}, {\i ODBC} or {\i UniDAC}) are able to have a huge boost at data insertion.
{\i SQlite3, MySQL, PostgreSQL, MSSQL 2008, DB2, Informix} and {\i NexusDB} handle {\f1\fs20 INSERT} statements with multiple {\f1\fs20 VALUES}, in the following SQL-92 standard syntax, using parameters:
$INSERT INTO TABLE (column-a, [column-b, ...])
$VALUES ('value-1a', ['value-1b', ...]),
$       ('value-2a', ['value-2b', ...]),
$       ...
{\i Oracle} implements the weird-but-similar syntax (note the mandatory {\f1\fs20 SELECT} at the end):
$INSERT ALL
$  INTO phone_book VALUES ('John Doe', '555-1212')
$  INTO phone_book VALUES ('Peter Doe', '555-2323')
$SELECT * FROM DUAL;
{\i Firebird} implements its own syntax:
$execute block
$as
$begin
$  INSERT INTO phone_book VALUES ('John Doe', '555-1212');
$  INSERT INTO phone_book VALUES ('Peter Doe', '555-2323');
$end
As a result, most engines show a nice speed boost when using the {\f1\fs20 BatchAdd()} method. See @59@ for numbers and details.
If you want to use a {\i @*map/reduce@} algorithm in your application, or the @100@ - in addition to ORM data access - all those enhancements will speed up a lot your data process. Reading and writing huge amount of data has never been so fast and easy: it is time to replace stored-procedure process by high-level code implemented in your {\i Domain} service.
\page
:39 CRUD level cache
Starting with revision 1.16 of the framework, tuned record @*cache@ has been implemented at the @*CRUD@/@*REST@ful level, for specific tables or records, on both the {\i server} and {\i client} sides.
See @38@ for the other data cache patterns available in the framework, mainly @37@ at {\i SQlite3} level, on the server side. All {\i mORMot}'s data caches are using @2@ as storage format, which was found to be simple and efficient for this purpose.
:  Where to cache
In fact, a unique caching mechanism is available at the {\f1\fs20 TSQLRest} level, for both {\f1\fs20 TSQLRestClient} and {\f1\fs20 TSQLRestServer} kind of classes. Therefore, {\i Delphi} clients can have their own cache, and the Server can also have its own cache. A client without any cache (e.g. a rough AJAX client) will take advantage of the server cache, at least.
By default, there is no caching at REST level. Then you can use the {\f1\fs20 TSQLRest.Cache} property to tune your cache policy for each {\f1\fs20 TSQLRest} instance.
\graph mORMotCaching CRUD caching in mORMot
\Internet (VPN)\Local Network
node [shape=box];
\Local Network\Server
subgraph cluster_0 {
"Client 1\n(Delphi)";
"client\ncache";
label="PC 1";
}
subgraph cluster_1 {
"Client 2\n(AJAX)";
label="PC 2";
}
subgraph cluster_2 {
\Server\DB
"server\ncache";
label="PC Server";
}
subgraph cluster_3 {
"Client n\n(Delphi)";
"client\n cache";
label="PC n";
}
\Client 1¤(Delphi)\Local Network\JSON + REST¤over HTTP/1.1
\Client 2¤(AJAX)\Internet (VPN)\JSON + REST¤over HTTP/1.1
\Client n¤(Delphi)\Internet (VPN)
\client¤cache\Client 1¤(Delphi)
\Client n¤(Delphi)\client¤ cache
\client¤ cache\Client n¤(Delphi)
\Client 1¤(Delphi)\client¤cache
\server¤cache\Server
\Server\server¤cache
\Server\external DB
\
When caching is set {\i on the server} for a particular record or table, in-memory values could be retrieved from this cache instead of calling the database engine each time. When properly used, this will increase global server responsiveness and allow more clients to be served with the same hardware.
{\i On the client} side, a local in-memory cache could be first checked when a record is to be retrieved. If the item is found, the client uses this cached value. If the data item is not in the local cache, the query is then sent to the server, just as usual. Due to the high latency of a remote client-server request, adding caching on the client side does make sense. Client caching properties can be tuned in order to handle properly remote HTTP access via the Internet, which may be much slower than a local Network.
Our caching implementation is transparent to the CRUD code. The very same usual ORM methods are to be called to access a record ({\f1\fs20 Retrieve Update Add}), then either client or server cache will be used, if available. For applications that frequently access the same data - a large category - record-level caching improves both performance and scalability.
:  When to cache
The main problem with cache is about data that both changes and is accessed simultaneously by multiple clients.
In the current implementation, a "pessimistic" concurrency control is used by our framework, relying on explicit locks, and (ab)use of its @15@ general design. It is up to the coder to ensure that no major confusion could arise from concurrency issues.
You must tune caching at both Client and Server level - each side will probably require its own set of cache options.
In your project implementation, caching should better not to be used at first, but added on need, when performance and efficiency was found to be required. Adding a cache shall imply having automated regression tests available, since in a Client-Server multi-threaded architecture, "{\i premature optimization is the root of all evil}" (Donald Knuth).
The main rules may be simply:
- {\i Not to cache if it may break something relevant} (like a global monetary balance value);
- {\i Not to cache unless you need to} (see Knuth's wisdom);
- {\i Ensure that caching is worth it} (if a value is likely to be overridden often, it could be even slower to cache it);
- Test once, test twice, always test and do not forget to test even more.
In practice, caching issues could be difficult to track. So in case of doubt (why was this data not accurate? it sounds like an old revision?), you may immediately disable caching, then ensure that you were not too optimistic about your cache policy.
:  What to cache
Typical content of these two tuned caches can be any global configuration settings, or any other kind of unchanging data which is not likely to vary often, and is accessed simultaneously by multiple clients, such as catalog information for an on-line retailer.
Another good use of caching is to store data that changes but is accessed by only one client at a time. By setting a cache at the client level for such content, the server won't be called often to retrieve the client-specific data. In such case, the problem of handling concurrent access to the cached data doesn't arise.
Profiling can be necessary to identify which data is to be registered within those caches, either at the client and/or the server side. The logging feature - see @73@ - integrated to {\i mORMot} can be very handy to tune the caching settings, due to its unique customer-side profiling ability.
But most of the time, an human guess at the business logic level is enough to set which data is to be cached on each side, and ensure content coherency.
:  How to cache
A tuned caching mechanism can be defined, for both {\f1\fs20 TSQLRestClient} and {\f1\fs20 TSQLRestServer} classes, at ID level.
By default, REST level cache is disabled, until you call {\f1\fs20 TSQLRest.Cache}'s {\f1\fs20 SetCache()} and {\f1\fs20 SetTimeOut()} methods. Those methods will define the caching policy, able to specify which table(s) or record(s) are to be cached, either at the client or the server level.
Once enabled for a table and a set of IDs on a given table, any further call to {\f1\fs20 TSQLRest.Retrieve(aClass,aID)} or {\f1\fs20 TSQLRecord.Create(aRest,aID)} will first attempt to retrieve the {\f1\fs20 TSQLRecord} of the given {\f1\fs20 aID} from the internal {\f1\fs20 TSQLRestCache} instance's in-memory cache, if available.\line Note that more complex requests, like queries on other fields than the ID @*primary key@, or JOINed queries, won't be cached at REST level. But such requests may benefit of the @37@, at {\i SQLite3} level, on the server side.
For instance, here is how the Client-side caching is tested about one individual record:
!    (...)
!!    Client.Cache.SetCache(TSQLRecordPeople); // cache whole table
!    TestOne;
!!    Client.Cache.Clear; // reset cache settings
!!    Client.Cache.SetCache(Rec); // cache one record
!    // same as Client.Cache.SetCache(TSQLRecordPeople,Rec.ID);
!    TestOne;
!    (...)
!!    Database.Cache.SetCache(TSQLRecordPeople); // server-side
!    (...)
In the above code, {\f1\fs20 Client.Cache.Clear} is used to reset all cache settings (i.e. not only flush the cache content, but delete all settings previously made with {\f1\fs20 Cache.SetCache()} or {\f1\fs20 Cache.SetTimeOut()} calls. So in the above code, a global cache is first enabled for the whole {\f1\fs20 TSQLRecordPeople} table, then the cache settings are reset, then cache is enabled for only the particular {\f1\fs20 Rec} record.\line To reset the cache content (e.g. if you consider some values may be deprecated), just call the {\f1\fs20 Cache.Flush} methods (able to flush the in-memory cache for all tables, a given table, or a given record).
It's worth warning once again that it's up to the code responsibility to ensure that these caches are consistent over the network. Server side and client side have their own coherency profile to be ensured. The caching policy has to match your data model, and application use cases.
{\i On the Client side}, only local CRUD operations are tracked. According to the @*stateless@ design, adding a time out value does definitively make sense, unless the corresponding data is known to be dedicated to this particular client (like a @*session@ data). If no time out period is set, it's up to the client to flush its own cache on purpose, by using {\f1\fs20 TSQLRestClient.Cache.Flush()} methods.
{\i On the Server side}, all CRUD operations of the @*ORM@ (like {\f1\fs20 Add / Update / Delete}) will be tracked, and cache will be notified of any data change. But direct SQL statements changing table contents (like a {\f1\fs20 UPDATE} or a {\f1\fs20 DELETE} over one or multiple rows with a {\f1\fs20 WHERE} clause) are not tracked by the current implementation: in such case, you'll have to manually flush the server cache content, to enforce data coherency. If such statements did occur on the server side, {\f1\fs20 TSQLRestServer.Cache.Flush()} methods are to be called, e.g. in the services which executed the corresponding SQL. If such non-CRUD statements did occur on the client side, it is possible to ensure that the server content is coherent with the client side, via a dedicated {\f1\fs20 TSQLRestClientURI.ServerCacheFlush()} method, which will call a dedicated standard service on the server to flush its cache content on purpose.
:201  Business logic and API cache
If your implementation follows a good design - see @155@ - the high-level logic is encapsulated into business types, and you won't use directly the {\f1\fs20 TSQLRecord} definitions. Another good practice is to define @*DTO@ types - see @156@ - probably as {\i records} or {\i dynamic arrays}.
The best performance will be achieved if the data is already known by the service, and returned immediately. Even if our @*ORM@ is very fast - thanks to its diverse cache levels we just wrote about - it may be hosted in another service, so a network delay may occur. The less communication, the better.
You may consider using @200@ instances over your business objects, or your DTO objects. You may start with no cache in the business or application layers, but once some bottlenecks are identified - e.g. by carefully looking at the logs generated by the framework @73@, defining some {\f1\fs20 @*TSynDictionary@} instances could help a lot. To release memory, don't forget to setup a proper {\f1\fs20 TimeOutSeconds} value.
:23Server side SQL/ORM process
%cartoon02.png
In your developer background and history, you may have been used to write your business code as {\i @**stored procedure@s}, to be executed on the server side.\line In short, a {\i stored procedure} is a way of moving some data-intensive SQL process on the database side. A client will ask for some data to be retrieved or processed on the server, and all actions will be taken on the server: since no data has to be exchanged between the client and the server, such a feature is usually much faster than a pure client-sided solution.
Since {\i mORMot} is {\i Client/Server} from the ground up, it features some unique ways of improving data-intensive process on the client or server sides, without necessary relying on proprietary {\i stored procedures}.
This chapter is worth reading, if you start a new {\i mORMot} project, and wonder about the architecture of your upcoming applications, or if you are integrating a {\i mORMot} server in an existing application... in which you or your predecessors may have (ab)used of stored procedures.\line It is time to sit down first, and take counsel how your project may be optimized enough to scale and profit.
\page
: Optimize for performance
So, let's do it the {\i mORMot}'s way.
As we discussed, the main point about {\i stored procedures} is performance. But they are not magic bullet either: we all have seen slow and endless process in {\i stored procedures}, almost killing a database server in production. Just as with regular client-side process.
And don't be fooled by performance: {\i make it right, then make it fast}.\line We could make ourself a motto of this Martin Fowler's remark:
{\i One of the first questions people consider with this kind of thing is performance. Personally I don't think performance should be the first question. My philosophy is that most of the time you should focus on writing maintainable code. Then use a profiler to identify hot spots and then replace only those hot spots with faster but less clear code. The main reason I do this is because in most systems only a very small proportion of the code is actually performance critical, and it's much easier to improve the performance of well factored maintainable code.}
See @http://www.martinfowler.com/articles/dblogic.html - nice link by the way, if you want to identify some best practice about implementing a persistence layer for our business code.
If you are using a {\i mORMot} server for the first time, you may be amazed by how most common process will sound just immediate. You can capitalize on the framework optimizations, which are able to unleash the computing power of your hardware, then refine your code only when performance matters.
In order to speed up our data processing, we first have to consider the classic architecture of a {\i mORMot} application - see @%%mORMotDesign3@.\line A {\i mORMot} client will have two means of accessing its data:
- Either from CRUD / ORM methods;
- Or via services, most probably {\i interface-based services} - see @63@.
Our optimization goals will therefore be leaded into those two directions.
:  Profiling your application
If you worry about performance, first reflex may be to enable the framework logging, even on customer sites.
The profiling abilities of the {\f1\fs20 TSynLog} class, used everywhere in our framework, will allow you to identify the potential bottlenecks of your client applications. See @16@ about this feature.\line From our experiment, we can assure you that the first time you setup {\i mORMot} advanced logging and profiling on a real application, you may find issues you may never have think of by yourself.
Fight against any duplicated queries, unnecessary returned information (do not forget to specify the fields to be retrieved to your {\f1\fs20 CreateAndFillPrepare()} request), unattended requests triggered by the User Interface (typically a {\f1\fs20 TEdit}'s {\f1\fs20 OnChange} event may benefit of using a {\f1\fs20 TTimer} before asking for auto-completion)...
Once you have done this cleaning, you may be proud of you, and it may be enough for your customers to enjoy your application. You deserve a coffee or a drink.
:  Client-side caching
You may have written most of your business logic on the client side, and use CRUD / ORM methods to retrieve the information from your {\i mORMot} server.\line This is perfectly valid, but may be slow, especially if a lot of individual requests are performed over the network, which may be with high latency (e.g. over the Internet).
A new step of optimization, once you did identify your application bottlenecks via profiling, may be to tune your ORM client cache. In fact, there are several layers of caching available in a {\i mORMot} application. See @39@ for details about these features.\line Marking some tables as potentially cached on the client side may induce a noticeable performance boost, with no need of changing your client code.
:  Write business logic as Services
A further step of optimization may be to let the business logic be processed on the server side, within a @*service@.\line In fact, you will then switch your mind from a classic @7@ to a @17@.
As a result, your process may take much less time, and you may also be able to benefit from some other optimization tricks, like dedicated caching in your service.\line For instance, consider writing your service in {\f1\fs20 sicShared} mode instead of the default {\f1\fs20 sicSingle} mode - see @92@ - and let some intangible objects be stored as implementation class fields: the next request from a client will not need to load this data from the database, but instead retrieve the information directly from memory, with no latency.\line You may also consider using {\f1\fs20 sicClientDriven} mode, and cache some client-specific information as implementation class fields.
Beside optimization, your code will probably become easier to maintain and scale, when running as services. SOA is indeed a very convenient pattern, and will induce nice side effects, like the ability to switch to multi-platform clients, including mobile or AJAX, since your business logic will stay on the application server.
:  Using ORM full power
On the server side, your business code, written using CRUD / ORM methods, could be optimized.
First of all, ORM caching may also be used. Any unneeded round-trip to the database - even more with @27@ - could impact your application responsiveness. Then your business logic, written as services, will benefit from it.
Then you may regroup all your database modifications using @28@.\line This will offer several benefits:
- Transaction support (nothing is written to the database until {\f1\fs20 BatchSend} method is executed) similar to the @100@;
- Faster insertion, update or deletion - via @78@ and @99@;
- Perfect integration with the ORM.
In fact, we found out that {\i Array DML} or {\i optimized INSERT} could be much faster than a regular {\i stored procedure}, with individual SQL statements run in a loop.
: Stored procedures
:101  Why to avoid stored procedures
In practice, {\i stored procedures} have some huge drawbacks:
- Your business logic is tied to the data layout used for storage - and the {\i Relational Model} is far away from natural language - see @91@;
- Debugging is somewhat difficult, since stored procedures will be executed on the database server, far away from your application;
- Each developer will probably need its own database instance to be able to debug its own set of {\i stored procedures};
- Project deployment becomes complex, since you have to synchronize your application and database server;
- Cursors and temporary tables, as commonly used in {\i stored procedures}, may hurt performance;
- They couple you with a particular database engine: you are tied to use {\i Java}, {\i C#} or a P/SQL variant to write your business code, then switching from {\i @*Oracle@} to {\i @*PostgreSQL@} or {\i @*MS SQL@} will be error prone, if not impossible;
- They may consume some precious hardware resources on your database server, which may be limited (e.g. proprietary engines like {\i Oracle} or {\i MS SQL} will force you to use only one CPU or a limited amount of RAM, unless you need to spend a lot of money to increase your license abilities);
- You will probably have limitations in the virtual environment running in your database engine: deprecated VM or libraries, restricted access to files or network due to security requirements, missing libraries;
- Inefficiency of parameters passing, especially when compared with class OOP programming - you are back to the procedural mode of the 80s;
- Parameters passing will probably result in sub-optimal SQL statements, handling all passed values even if not used;
- Flat design of stored procedures interfaces, far away from the interface segregation principle - see @160@;
- Let several versions of your business logic coexist on the same server is a nightmare to maintain;
- Unit testing is made difficult, since you won't be able to mock or stub - see @62@ - your {\i stored procedures} or your data;
- No popular SQL engine does allow {\i stored procedures} to be written in Delphi, so you won't be able to share code with your other projects;
- If you use an ORM in your main application, you need to manually maintain the table schema used in your stored procedures in synch with your object model - so you are loosing most of ORM benefits;
- What if you want to switch to {\i @*NoSQL@} storage, or a simple stand-alone version of your application?
We do not want to say, dogmatically, that {\i stored procedures} are absolute evil. Of course, you are free to use them, even with {\i mORMot}.\line All we wanted to point out is the fact that they are perhaps not the best fit with the design we would like to follow.
:  Stored procedures, anyway
There may be some cases where this ORM point of view, may be not enough for your project.\line Do not worry, as usual {\i mORMot} will allow you to do what you need.
The Server-Side services - see @49@ and @63@ - appear to be the more @*REST@ful compatible way of implementing a {\i @*stored procedure@} mechanism in our framework, then consume them from a {\i mORMot} client.
According to the current state of our framework, there are several ways of handling such a server-side SQL/ORM process:
- Write your own @*SQL function@ to be used in {\i @*SQLite3@} WHERE statements;
- Low-level dedicated {\i Delphi} stored procedures;
- External databases stored procedures.
We will discuss those options.\line The first two will in fact implement two types of "@*stored procedure@" at SQL level in pure {\i Delphi} code, making our {\i SQlite3} kernel as powerful as other @*Client-Server@ RDBMS solutions. The latest option may be considered, especially when moving from legacy applications, still relying on stored procedures for their business logic.
:22   Custom SQL functions
The {\i @*SQLite3@} engine defines some standard @**SQL function@s, like {\f1\fs20 abs() min() max()} or {\f1\fs20 upper()}. A complete list is available at @http://www.sqlite.org/lang_corefunc.html
One of the greatest {\i SQLite3} feature is the ability to define custom SQL functions in high-level language. In fact, its C API allows implementing new functions which may be called within a SQL query. In other database engine, such functions are usually named UDF (for {\i User Defined Functions}).
Some custom already defined SQL functions are defined by the framework.\line You may have to use, on the Server-side:
- {\f1\fs20 Rank} used for page ranking in @24@;
- {\f1\fs20 Concat} to process fast string concatenation;
- {\f1\fs20 Soundex SoundexFR SoundexES} for computing the English / French / Spanish soundex value of any text;
- {\f1\fs20 @*IntegerDynArrayContains@, ByteDynArrayContains, WordDynArrayContains, CardinalDynArrayContains, Int64DynArrayContains, CurrencyDynArrayContains, RawUTF8DynArrayContainsCase, RawDynArrayContainsNoCase} for direct search inside a BLOB column containing some @*dynamic array@ binary content (expecting either an INTEGER or a TEXT search value as 2nd parameter).
Those functions are no part of the {\i SQlite3} engine, but are available inside our ORM to handle BLOB containing @*dynamic array@ properties, as stated in @21@.
Since you may use such SQL functions in an {\f1\fs20 UPDATE} or {\f1\fs20 INSERT} SQL statement, you may have an easy way of implementing server-side process of complex data, as such:
$ UPDATE MyTable SET SomeField=0 WHERE IntegerDynArrayContains(IntArrayField,:(10):)
:    Implementing a function
Let us implement a {\f1\fs20 CharIndex()} SQL function, defined as such:
! CharIndex ( SubText, Text [ , StartPos ] )
In here, {\f1\fs20 SubText} is the string of characters to look for in {\f1\fs20 Text}. {\f1\fs20 StartPos} indicates the starting index where {\f1\fs20 charindex()} should start looking for {\f1\fs20 SubText} in {\f1\fs20 Text}. Function shall return the position where the match occurred, 0 when no match occurs. Characters are counted from 1, just like in {\f1\fs20 PosEx()} {\i Delphi} function.
The SQL function implementation pattern itself is explained in the {\f1\fs20 sqlite3.create_function_v2()} and {\f1\fs20 TSQLFunctionFunc}:
- {\f1\fs20 argc} is the number of supplied parameters, which are available in {\f1\fs20 argv[]} array (you can call {\f1\fs20 ErrorWrongNumberOfArgs(Context)} in case of unexpected incoming number of parameters);
- Use {\f1\fs20 sqlite3.value_*(argv[*])} functions to retrieve a parameter value;
- Then set the result value using {\f1\fs20 sqlite3.result_*(Context,*)} functions.
Here is typical implementation code of the {\f1\fs20 CharIndex()} SQL function, calling the expected low-level {\i SQLite3} API (note the {\b {\f1\fs20 cdecl}} calling convention, since it is a {\i SQLite3} / C callback function):
!procedure InternalSQLFunctionCharIndex(Context: TSQLite3FunctionContext;
!  argc: integer; var argv: TSQLite3ValueArray); cdecl;
!var StartPos: integer;
!begin
!  case argc of
!  2: StartPos := 1;
!  3: begin
!!    StartPos := sqlite3.value_int64(argv[2]);
!    if StartPos<=0 then
!      StartPos := 1;
!  end;
!  else begin
!    ErrorWrongNumberOfArgs(Context);
!    exit;
!  end;
!  end;
!  if (sqlite3.value_type(argv[0])=SQLITE_NULL) or
!     (sqlite3.value_type(argv[1])=SQLITE_NULL) then
!    sqlite3.result_int64(Context,0) else
!!    sqlite3.result_int64(Context,SynCommons.PosEx(
!!      sqlite3.value_text(argv[0]),sqlite3.value_text(argv[1]),StartPos));
!end;
This code just get the parameters values using {\f1\fs20 sqlite3.value_*()} functions, then call the {\f1\fs20 PosEx()} function to return the position of the supplied text, as an INTEGER, using {\f1\fs20 sqlite3.result_int64()}.
The local {\f1\fs20 StartPos} variable is used to check for an optional third parameter to the SQL function, to specify the character index to start searching from.
The special case of a {\f1\fs20 NULL} parameter is handled by checking the incoming argument type, calling {\f1\fs20 sqlite3.value_type(argv[])}.
:    Registering a function
:     Direct low-level SQLite3 registration
Since we have a {\f1\fs20 InternalSQLFunctionCharIndex()} function defined, we may register it with direct {\i SQLite3} API calls, as such:
!  sqlite3.create_function_v2(Demo.DB,
!    'CharIndex',2,SQLITE_ANY,nil,InternalSQLFunctionCharIndex,nil,nil,nil);
!  sqlite3.create_function_v2(Demo.DB,
!    'CharIndex',3,SQLITE_ANY,nil,InternalSQLFunctionCharIndex,nil,nil,nil);
The function is registered twice, one time with 2 parameters, then with 3 parameters, to add an overloaded version with the optional {\f1\fs20 StartPos} parameter.
:     Class-driven registration
It is possible to add some custom SQL functions to the {\i SQlite3} engine itself, by creating a {\f1\fs20 TSQLDataBaseSQLFunction} custom class and calling the {\f1\fs20 TSQLDataBase.RegisterSQLFunction} method.
The standard way of using this is to override the {\f1\fs20 @*TSQLRestServerDB@.InitializeEngine virtual} method, calling {\f1\fs20 DB.RegisterSQLFunction()} with an defined {\f1\fs20 TSQLDataBaseSQLFunction} custom class.
So instead of calling low-level {\f1\fs20 sqlite3.create_function_v2()} API, you can declare the {\f1\fs20 CharIndex} SQL function as such:
!  Demo.RegisterSQLFunction(InternalSQLFunctionCharIndex,2,'CharIndex');
!  Demo.RegisterSQLFunction(InternalSQLFunctionCharIndex,3,'CharIndex');
The two lines above will indeed wrap the following code:
!  Demo.RegisterSQLFunction(TSQLDataBaseSQLFunction.Create(InternalSQLFunctionCharIndex,2,'CharIndex'));
!  Demo.RegisterSQLFunction(TSQLDataBaseSQLFunction.Create(InternalSQLFunctionCharIndex,3,'CharIndex'));
The {\f1\fs20 RegisterSQLFunction()} method is called twice, one time with 2 parameters, then with 3 parameters, to add an overloaded version with the optional {\f1\fs20 StartPos} parameter, as expected.
:     Custom class definition
The generic function definition may be completed, in our framework, with a custom class definition, which is handy to have some specific context, not only relative to the current SQL function context, but global and static to the whole application process.
\graph HierTSQLDataBaseSQLFunctionDynArray TSQLDataBaseSQLFunction classes hierarchy
\TSQLDataBaseSQLFunction\TObject
\TSQLDataBaseSQLFunctionDynArray\TSQLDataBaseSQLFunction
\
For instance, the following method will register a SQL function able to search into a BLOB-stored custom dynamic array type:
!procedure TSQLDataBase.RegisterSQLFunction(aDynArrayTypeInfo: pointer;
!  aCompare: TDynArraySortCompare; const aFunctionName: RawUTF8);
!begin
!  RegisterSQLFunction(
!    TSQLDataBaseSQLFunctionDynArray.Create(aDynArrayTypeInfo,aCompare,aFunctionName));
!end;
We specify directly the {\f1\fs20 TSQLDataBaseSQLFunctionDynArray} class instance to work with, which adds two needed protected fields to the {\f1\fs20 TSQLDataBaseSQLFunction} root class:
- A {\f1\fs20 fDummyDynArray TDynArray} instance which will handle the dynamic array RTTI handling;
- A {\f1\fs20 fDummyDynArrayValue pointer}, to be used to store the dynamic array reference values to be used during the dynamic array process.
Here is the corresponding class definition:
!  /// to be used to define custom SQL functions for dynamic arrays BLOB search
!  TSQLDataBaseSQLFunctionDynArray = class(TSQLDataBaseSQLFunction)
!  protected
!    fDummyDynArray: TDynArray;
!    fDummyDynArrayValue: pointer;
!  public
!    /// initialize the corresponding SQL function
!    // - if the function name is not specified, it will be retrieved from the type
!    // information (e.g. TReferenceDynArray will declare 'ReferenceDynArray')
!    // - the SQL function will expect two parameters: the first is the BLOB
!    // field content, and the 2nd is the array element to search (set with
!    // TDynArray.ElemSave() or with BinToBase64WithMagic(aDynArray.ElemSave())
!    // if called via a Client and a JSON prepared parameter)
!    // - you should better use the already existing faster SQL functions
!    // Byte/Word/Integer/Cardinal/Int64/CurrencyDynArrayContains() if possible
!    // (this implementation will allocate each dynamic array into memory before
!    // comparison, and will be therefore slower than those optimized versions)
!    constructor Create(aTypeInfo: pointer; aCompare: TDynArraySortCompare;
!      const aFunctionName: RawUTF8=''); override;
!  end;
And the constructor implementation:
!constructor TSQLDataBaseSQLFunctionDynArray.Create(aTypeInfo: pointer;
!  aCompare: TDynArraySortCompare; const aFunctionName: RawUTF8);
!begin
!  fDummyDynArray.Init(aTypeInfo,fDummyDynArrayValue);
!  fDummyDynArray.Compare := aCompare;
!  inherited Create(InternalSQLFunctionDynArrayBlob,2,aFunctionName);
!end;
The {\f1\fs20 InternalSQLFunctionDynArrayBlob} function is a low-level {\i SQlite3} engine SQL function prototype, which will retrieve a BLOB content, then un-serialize it into a dynamic array (using the {\f1\fs20 fDummyDynArrayValue. LoadFrom} method), then call the standard {\f1\fs20 ElemLoadFind} method to search the supplied element, as such:
! (...)
!  with Func.fDummyDynArray do
!  try
!!    LoadFrom(DynArray); // temporary allocate all dynamic array content
!    try
!!      if ElemLoadFind(Elem)<0 then
!        DynArray := nil;
!    finally
!      Clear; // release temporary array content in fDummyDynArrayValue
!    end;
! (...)
You can define a similar class in order to implement your own custom SQL function.
Here is how a custom SQL function using this {\f1\fs20 TSQLDataBaseSQLFunctionDynArray} class is registered in the supplied unitary tests to an existing database connection:
!  Demo.RegisterSQLFunction(TypeInfo(TIntegerDynArray),SortDynArrayInteger,
!    'MyIntegerDynArrayContains');
This new SQL function expects two BLOBs arguments, the first being a reference to the BLOB column, and the 2nd the searched value. The function can be called as such (lines extracted from the framework regression tests):
!    aClient.OneFieldValues(TSQLRecordPeopleArray,'ID',
!      FormatUTF8('MyIntegerDynArrayContains(Ints,:("%"):)',
!        [BinToBase64WithMagic(@k,sizeof(k))]),IDs);
Note that since the 2nd parameter is expected to be a BLOB representation of the searched value, the {\f1\fs20 BinToBase64WithMagic} function is used to create a BLOB parameter, as expected by the ORM. Here, the element type is an {\f1\fs20 integer}, which is a pure binary variable (containing no reference-counted internal fields): so we use direct mapping from its binary in-memory representation; for more complex element type, you should use the generic {\f1\fs20 BinToBase64WithMagic(aDynArray.ElemSave())} expression instead, calling {\f1\fs20 TDynArray. ElemSave} method.
Note that we did not use here the overloaded {\f1\fs20 OneFieldValues} method expecting '?' bound parameters here, but we may have use it as such:
!    aClient.OneFieldValues(TSQLRecordPeopleArray,'ID',
!      FormatUTF8('MyIntegerDynArrayContains(Ints,?)',[],
!        [BinToBase64WithMagic(@k,sizeof(k))]),IDs);
Since the {\f1\fs20 MyIntegerDynArrayContains} function will create a temporary dynamic array in memory from each row (stored in {\f1\fs20 fDummyDynArrayValue}), the dedicated {\f1\fs20 IntegerDynArrayContains} SQL function is faster.
:   Low-level SQLite3 stored procedure in Delphi
To implement a more complete request, and handle any kind of stored data in a column (for instance, some TEXT format to be parsed), a {\f1\fs20 TOnSQLStoredProc} event handler can be called for every row of a prepared statement, and is able to access directly to the database request.\line This event handler can be specified to the {\f1\fs20 @*TSQLRestServerDB@.StoredProcExecute()} method.\line Be aware that code inside this event handler should not use the @*ORM@ methods of the framework, but direct low-level {\i SQLite3} access (to avoid re-entrance issues).
This will allow direct content modification during the {\f1\fs20 SELECT} statement. Be aware that, up to now, @20@ {\f1\fs20 TSQLVirtualTableCursorJSON} cursors are not safe to be used if the {\i Virtual Table} data is modified.
See the description of the {\f1\fs20 TOnSQLStoredProc} event handler and associated {\f1\fs20 StoredProcExecute()} method in the second part of this document.
:   External stored procedure
If the application relies on external databases - see @27@ - the external database may be located on a remote computer.
In such situation, all RESTful Server-sided solutions could produce a lot of network traffic. In fact, custom SQL functions or stored procedures both use the {\i @*SQLite3@} engine as root component.
In order to speed up the process, you may define some RDMS stored procedures in the external database syntax (P/SQL, {\i .Net}, {\i Java} or whatever), then define some @11@ to launch those functions.\line Note that in this case, you'll loose the database independence of the framework, and most of the benefits of using an ORM/ODM - later on, switching to another database engine may become impossible. Such RDBMS stored procedures may be envisaged only during the transition phase of an existing application. @28@ has almost all the speed advantages of stored procedures, with the benefit of a pure object oriented code, easy to debug and maintain.
\page
:11 Server side Services
%cartoon03.png
In order to follow a @17@ design, your application's business logic can be implemented in several ways using {\i mORMot}:
- Via some {\f1\fs20 @*TSQLRecord@} inherited classes, inserted into the database {\i model}, and accessible via some @*REST@ful URI - this is implemented by our @*ORM@ architecture - see @35@;
- By some RESTful @**service@s, implemented in the Server as {\i published methods}, and consumed in the Client via native {\i Delphi} methods;
- Defining some RESTful {\i service @*contract@s} as standard {\i Delphi} {\f1\fs20 interface}, and then run it seamlesly on both client and client sides.
The first is similar to {\i RemObject's DataAbstract} product, which allows remote access to database, over several protocols. There are some similarities with {\i mORMot} (like on-the-fly SQL translation for external databases), but also a whole diverse use case (RAD/components and wizards versus ORM/MVC) and implementation ({\i mORMot} takes advantages of the {\i @*SQLite3@} SQL core and is much more optimized for speed and scaling).
If you paid for a {\i Delphi Architect} edition, the first two items can be compared to the {\i DataSnap} Client-Server features. Since {\i Delphi} 2010, you can in fact define @*JSON@-based RESTful services, in addition to the original {\i DCOM/DBExpress} remote data broker. It makes uses of the new RTTI available since {\i Delphi} 2010, but it has some known stability and performance issues, and lack of strong security. It is also RAD/Wizard based, whereas {\i mORMot} uses a code approach.
The last item is purely interface-based, so matches the "designed by contract" principle - see @47@ - as implemented by Microsoft's @*WCF@ technology - see @65@. We included most of the nice features made available in WCF in {\i mORMot}, in a @*KISS@ {\i @*convention over configuration@} manner.
So {\i mORMot} is quite unique, in the fact that it features, in one unique code base, all three ways of implementing a @*SOA@ application. And it is an Open Source project, existing since years - you won't be stucked with proprietary code nor licenses. You can move your existing code base into a {\i Domain-Driven Design}, on your management pace (and money), without the need of upgrading to the latest version of the IDE.
:49Client-Server services via methods
%cartoon04.png
To implement a service in the {\i Synopse mORMot framework}, the first method is to define @**published method@ Server-side, then use easy functions about JSON or URL-parameters to get the request encoded and decoded as expected, on Client-side.
We'll implement the same example as in the official Embarcadero docwiki page above. Add two numbers. Very useful service, isn't it?
: Publishing a service on the server
On the server side, we need to customize the standard {\f1\fs20 TSQLRestServer} class definition (more precisely a {\f1\fs20 @*TSQLRestServerDB@} class which includes a {\i @*SQlite3@} engine, or a lighter {\f1\fs20 @*TSQLRestServerFullMemory@} kind of server, which is enough for our purpose), by adding a new {\f1\fs20 published} method:
!type
!  TSQLRestServerTest = class(TSQLRestServerFullMemory)
!   (...)
!!  published
!!    procedure Sum(Ctxt: TSQLRestServerURIContext);
!  end;
The method name ("Sum") will be used for the URI encoding, and will be called remotely from {\i ModelRoot/Sum} URL. The {\i ModelRoot} is the one defined in the {\f1\fs20 Root} parameter of the {\i model} used by the application.
This method, like all Server-side methods, MUST have the same exact parameter definition as in the {\f1\fs20 TSQLRestServerCallBack} prototype, i.e. only one {\f1\fs20 Ctxt} parameter, which refers to the whole {\i execution context}:
!type
!  TSQLRestServerCallBack = procedure(Ctxt: TSQLRestServerURIContext) of object;
Then we implement this method:
!procedure TSQLRestServerTest.Sum(Ctxt: TSQLRestServerURIContext);
!begin
!  Ctxt.Results([Ctxt['a']+Ctxt['b']]);
!end;
The {\f1\fs20 Ctxt} variable publish some properties named {\f1\fs20 InputInt[] InputDouble[] InputUTF8[]} and {\f1\fs20 Input[]} able to retrieve directly a parameter value from its name, respectively as {\f1\fs20 Integer/Int64}, {\f1\fs20 @*double@}, {\f1\fs20 RawUTF8} or {\f1\fs20 variant}. The {\f1\fs20 Ctxt.Input[]} array property, returning {\f1\fs20 variant} values, has been defined as {\f1\fs20 default} array property for the {\f1\fs20 TSQLRestServerURIContext} class, so writing {\f1\fs20 Ctxt['a']} is the same as writing {\f1\fs20 Ctxt.Input['a']}.
Therefore, the code above using {\f1\fs20 Ctxt[]} or {\f1\fs20 Ctxt.Input[]} will introduce a conversion via a {\f1\fs20 variant}, which may be a bit slower, and in case of {\f1\fs20 string} content, may loose some content for older non Unicode versions of {\i Delphi}. So it is a good idea to use the exact expected {\f1\fs20 Input*[]} property corresponding to your value type. It does make sense even more when handling text, i.e. {\f1\fs20 InputUTF8[]} is to be used in such case. For our floating-point computation method, we may have coded it as such:
Those methods will raise an {\f1\fs20 EParsingException} exception if the parameter is not available at the URI. So you may want to use {\f1\fs20 InputExists[]} or even {\f1\fs20 InputIntOrVoid[] InputDoubleOrVoid[] InputUTF8OrVoid[] InputOrVoid[]} methods, which won't raise any exception but return a void value (i.e. either {\f1\fs20 0}, {\f1\fs20 ""} or {\f1\fs20 Unassigned}).
!procedure TSQLRestServerTest.Sum(Ctxt: TSQLRestServerURIContext);
!begin
!  with Ctxt do
!    Results([InputDouble['a']+InputDouble['b']]);
!end;
The {\f1\fs20 Ctxt.Results([])} method is used to return the service value as one JSON object with one {\f1\fs20 "Result"} member, with default @*MIME@-type {\f1\fs20 JSON_CONTENT_TYPE}.
For instance, the following request URI:
$ GET /root/Sum?a=3.12&b=4.2
will let our server method return the following JSON object:
$ {"Result":7.32}
That is, a perfectly AJAX-friendly request.
Note that all parameters are expected to be plain case-insensitive {\f1\fs20 'A'..'Z','0'..'9'} characters.
An {\i important point} is to remember that the implementation of the callback method {\b must be thread-safe} - as stated by @25@ and @184@. In fact, the {\f1\fs20 TSQLRestServer.URI} method expects such callbacks to handle the thread-safety on their side. It's perhaps some more work to handle a {\i critical section} in the implementation, but, in practice, it's the best way to achieve performance and scalability: the resource locking can be made at the tiniest code level.
\page
: Defining the client
The client-side is implemented by calling some dedicated methods, and providing the service name ({\f1\fs20 'sum'}) and its associated parameters:
!function Sum(aClient: TSQLRestClientURI; a, b: double): double;
!var err: integer;
!begin
!  val(aClient.CallBackGetResult('sum',['a',a,'b',b]),Result,err);
!end;
You could even implement this method in a dedicated client method - which make sense:
!type
!  TMyClient = class(TSQLHttpClient) // could be TSQLRestClientURINamedPipe
!  (...)
!    function Sum(a, b: double): double;
!  (...)
!
!function TMyClient.Sum(a, b: double): double;
!var err: integer;
!begin
!  val(CallBackGetResult('sum',['a',a,'b',b]),Result,err);
!end;
This later implementation is to be preferred on real applications.
You have to create the server instance, and the corresponding {\f1\fs20 TSQLRestClientURI} (or {\f1\fs20 TMyClient}), with the same database model, just as usual...
On the Client side, you can use the {\f1\fs20 CallBackGetResult} method to call the service from its name and its expected parameters, or create your own caller using the {\f1\fs20 UrlEncode()} function. Note that you can specify most class instance into its JSON representation by using some {\f1\fs20 TObject} into the method arguments:
!function TMyClient.SumMyObject(a, b: TMyObject): double;
!var err: integer;
!begin
!  val(CallBackGetResult('summyobject',['a',a,'b',b]),Result,err);
!end;
This Client-Server protocol uses JSON here, as encoded server-side via {\f1\fs20 Ctxt.Results()} method, but you can serve any kind of data, binary, HTML, whatever... just by overriding the content type on the server with {\f1\fs20 Ctxt.Returns()}.
\page
: Direct parameter marshalling on server side
We have used above the {\f1\fs20 Ctxt[]} and {\f1\fs20 Ctxt.Input*[]} properties to retrieve the input parameters. This is pretty easy to use and powerful, but the supplied {\f1\fs20 Ctxt} gives full access to the input and output context.
Here is how we may implement the fastest possible parameters parsing - see sample {\f1\fs20 Project06Server.dpr}:
!procedure TSQLRestServerTest.Sum(Ctxt: TSQLRestServerURIContext);
!var a,b: double;
!begin
!  if UrlDecodeNeedParameters(Ctxt.Parameters,'A,B') then begin
!    while Ctxt.Parameters<>nil do begin
!      UrlDecodeDouble(Ctxt.Parameters,'A=',a);
!      UrlDecodeDouble(Ctxt.Parameters,'B=',b,@Ctxt.Parameters);
!    end;
!    Ctxt.Results([a+b]);
!  end else
!    Ctxt.Error('Missing Parameter');
!end;
The only not obvious part of this code is the parameters marshalling, i.e. how the values are retrieved from the incoming {\f1\fs20 Ctxt.Parameters} text buffer, then converted into native local variables.
On the Server side, typical implementation steps are therefore:
- Use the {\f1\fs20 UrlDecodeNeedParameters} function to check that all expected parameters were supplied by the caller in {\f1\fs20 Ctxt.Parameters};
- Call {\f1\fs20 UrlDecodeInteger / UrlDecodeInt64 / UrlDecodeDouble / UrlDecodeExtended / UrlDecodeValue / UrlDecodeObject} functions (all defined in {\f1\fs20 SynCommons.pas}) to retrieve each individual parameter from standard JSON content;
- Implement the service (here it is just the {\f1\fs20 a+b} expression);
- Then return the result calling {\f1\fs20 Ctxt.Results()} method or {\f1\fs20 Ctxt.Error()} in case of any error.
The powerful {\f1\fs20 UrlDecodeObject} function (defined in {\f1\fs20 mORMot.pas}) can be used to un-serialize most class instance from its textual JSON representation ({\f1\fs20 @*TPersistent@, @*TSQLRecord@, TStringList}...).
Using {\f1\fs20 Ctxt.Results()} will encode the specified values as a JSON object with one {\f1\fs20 "Result"} member, with default mime-type {\f1\fs20 JSON_CONTENT_TYPE}:
$ {"Result":"OneValue"}
or a JSON object containing an array:
$ {"Result":["One","two"]}
\page
: Returns non-JSON content
Using {\f1\fs20 Ctxt.Returns()} will let the method return the content in any format, e.g. as a JSON object (via the overloaded {\f1\fs20 Ctxt.Returns([])} method expecting field name/value pairs), or any content, since the returned @**MIME@-type can be defined as a parameter to {\f1\fs20 Ctxt.Returns()} - it may be useful to specify another mime-type than the default constant {\f1\fs20 JSON_CONTENT_TYPE}, i.e. {\f1\fs20 'application/json; charset=UTF-8'}, and returns plain text, HTML or binary.
For instance, you can return directly a value as plain text:
!procedure TSQLRestServer.Timestamp(Ctxt: TSQLRestServerURIContext);
!begin
!  Ctxt.Returns(Int64ToUtf8(ServerTimestamp),HTTP_SUCCESS,TEXT_CONTENT_TYPE_HEADER);
!end;
Or you can return some binary file, retrieving the corresponding MIME type from its binary content:
!procedure TSQLRestServer.GetFile(Ctxt: TSQLRestServerURIContext);
!var fileName: TFileName;
!    content: RawByteString;
!    contentType: RawUTF8;
!begin
!  fileName :=  'c:\data\'+ExtractFileName(Ctxt['filename']); // or Ctxt.Input['filename']
!  content := StringFromFile(fileName);
!  if content='' then
!    Ctxt.Error('',HTTP_NOTFOUND) else
!    Ctxt.Returns(content,HTTP_SUCCESS,HEADER_CONTENT_TYPE+
!         GetMimeContentType(pointer(content),Length(content),fileName));
!end;
The corresponding client method may be defined as such:
!function TMyClient.GetFile(const aFileName: RawUTF8): RawByteString;
!begin
!  if CallBackGet('GetFile',['filename',aFileName],RawUTF8(result))<>HTTP_SUCCESS then
!    raise Exception.CreateFmt('Impossible to get file: %s',[result]);
!end;
Note that the {\f1\fs20 Ctxt.ReturnFile()} method - see @95@ - is preferred than manual file retrieval as implemented in this {\f1\fs20 TSQLRestServer.GetFile()} method. It is shown here for demonstration purposes only.
If you use HTTP as communication protocol, you can consume these services, implemented Server-Side in fast {\i Delphi} code, with any @*AJAX@ application on the client side.
Using {\f1\fs20 GetMimeContentType()} when sending non JSON content (e.g. picture, pdf file, binary...) will be interpreted as expected by any standard Internet browser: it could be used to serve some good old HTML content within a page, not necessary consume the service via {\i JavaScript} .
\page
: Advanced process on server side
On server side, method definition has only one {\f1\fs20 Ctxt} parameter, which has several members at calling time, and publish all service calling features and context, including {\i RESTful} URI routing, session handling or low-level HTTP headers (if any).
At first, {\f1\fs20 Ctxt} may indicate the expected {\f1\fs20 TSQLRecord} ID and {\f1\fs20 TSQLRecord} class, as decoded from {\i RESTful} URI. It means that a service can be related to any table/class of our @*ORM@ framework, so you will be able to create easily any RESTful compatible requests on URI like {\f1\fs20 ModelRoot/TableName/TableID/MethodName}. The ID of the corresponding record is decoded from its {\i RESTful} scheme into {\f1\fs20 Ctxt.TableID}, and the table is available in {\f1\fs20 Ctxt.Table} or {\f1\fs20 Ctxt.TableIndex} (if you need its index in the associated server Model).
For example, here we return a @*BLOB@ field content as hexadecimal, according to its {\f1\fs20 TableName/TableID}:
!procedure TSQLRestServerTest.DataAsHex(Ctxt: TSQLRestServerURIContext);
!var aData: TSQLRawBlob;
!begin
!  if (self=nil) or (Ctxt.Table<>TSQLRecordPeople) or (Ctxt.TableID<=0) then
!    Ctxt.Error('Need a valid record and its ID') else
!  if RetrieveBlob(TSQLRecordPeople,Ctxt.TableID,'Data',aData) then
!    Ctxt.Results([SynCommons.BinToHex(aData)]) else
!    Ctxt.Error('Impossible to retrieve the Data BLOB field');
!end;
A corresponding client method may be:
!function TSQLRecordPeople.DataAsHex(aClient: TSQLRestClientURI): RawUTF8;
!begin
!  Result := aClient.CallBackGetResult('DataAsHex',[],RecordClass,fID);
!end;
If @*authentication@ - see @18@ - is used, the current session, user and group IDs are available in {\f1\fs20 Session / SessionUser / SessionGroup} fields. If authentication is not available, those fields are meaningless: in fact, {\f1\fs20 Ctxt.Context.Session} will contain either 0 ({\f1\fs20 CONST_AUTHENTICATION_SESSION_NOT_STARTED}) if any @*session@ is not yet started, or 1 ({\f1\fs20 CONST_AUTHENTICATION_NOT_USED}) if authentication mode is not active. Server-side implementation can use the {\f1\fs20 TSQLRestServer.SessionGetUser} method to retrieve the corresponding user details (note that when using this method, the returned {\f1\fs20 @*TSQLAuthUser@} instance is a local thread-safe copy which shall be freed when done).
In {\f1\fs20 Ctxt.Call^} member, you can access low-level communication content, i.e. all incoming and outgoing values, including headers and message body. Depending on the transmission protocol used, you can retrieve e.g. HTTP header information. For instance, here is how you may access the client remote IP address and application {\f1\fs20 @*User-Agent@}, at lowest level:
! aRemoteIP := FindIniNameValue(pointer(Ctxt.Call.InHead),'REMOTEIP: ');
! aUserAgent := FindIniNameValue(pointer(Ctxt.Call.InHead),'USER-AGENT: ');
Of course, for those fields, it is much preferred to use the {\f1\fs20 Ctxt.RemoteIP} or {\f1\fs20 Ctxt.UserAgent} properties, which use an efficient cache.
\page
:96 Browser speed-up for unmodified requests
When used over a slow network (e.g. over the Internet), you can set the optional {\f1\fs20 Handle304NotModified} parameter of both {\f1\fs20 Ctxt.Returns()} and {\f1\fs20 Ctxt.Results()} methods to return the response body only if it has changed since last time.
In practice, result content will be hashed (using {\f1\fs20 @*crc32c@} algorithm, and fast SSE4.2 hardware instruction, if available) and in case of no modification will return "{\i 304 Not Modified}" status to the browser, without the actual result content. Therefore, the response will be transmitted and received much faster, and will save a lot of bandwidth, especially in case of periodic server pooling (e.g. for client screen refresh).
Note that in case of hash collision of the {\f1\fs20 crc32c} algorithm (we never did see it happen, but such a mathematical possibility exists), a false positive "not modified" status may be returned; this option is therefore unset by default, and should be enabled only if your client does not handle any sensitive accounting process, for instance.
Be aware that you should {\i disable authentication} for the methods using this {\f1\fs20 Handle304NotModified} parameter, via a {\f1\fs20 TSQLRestServer.ServiceMethodByPassAuthentication()} call. In fact, our @*REST@ful authentication - see @18@ - uses a per-URI signature, which change very often (to avoid men-in-the-middle attacks). Therefore, any browser-side caching benefit will be voided if authentication is used: browser internal cache will tend to grow for nothing since the previous URIs are deprecated, and it will be a cache-miss most of the time. But when serving some static content (e.g. HTML content, fixed JSON values or even UI binaries), this browser-side caching can be very useful.
This @*stateless@ @9@ model will enable several levels of caching, even using an external {\i Content Delivery Network} (@*CDN@) service. See @97@ for some potential hosting architectures, which may let your {\i mORMot} server scale to thousands of concurrent users, served around the world with the best responsiveness.
:95 Returning file content
Framework's HTTP server is able to handle returning a file as response to a method-based service.\line The @88@ is even able to serve the file content asynchronously from kernel mode, with outstanding performance.
You can use the {\f1\fs20 Ctxt.ReturnFile()} method to return a file directly.\line This method is also able to guess the MIME type from the file extension, and handle {\f1\fs20 HTTP_NOTMODIFIED = 304} process, if {\f1\fs20 Handle304NotModified} parameter is {\f1\fs20 true}, using the file time stamp.
Another possibility may be to use the {\f1\fs20 Ctxt.ReturnFileFromFolder()} method, which is able to efficiently return any file specified by its URI, from a local folder. It may be very handy to
return some static web content from a {\i mORMot} HTTP server.
:192 JSON Web Tokens (JWT)
{\i JSON Web Token} ({\f1\fs20 @**JWT@}) is an open standard ({\i RFC 7519}) that defines a compact and self-contained way for securely transmitting information between parties as a JSON object. This information can be verified and trusted because it is digitally signed. JWTs can be signed using a secret (with the HMAC algorithm) or a public/private key pair using RSA or ECDSA.\line They can be used for:
- @*Authentication@: including a {\f1\fs20 JWT} to any HTTP request allows {\i Single Sign On} user validation across different domains;
- Secure Information Exchange: a small amount of data can be stored in the JWT payload, and is digitally signed to ensure its provenance and integrity.
See @http://jwt.io for an introduction to {\i JSON Web Tokens}.
Our framework implements {\f1\fs20 JWT}:
- {\f1\fs20 "HS256/384/512"} (@*HMAC-SHA2@-256/384/512), {\f1\fs20 "ES256"} (256-bit @*ECDSA@) standard algorithms, and {\f1\fs20 "S3256/384/512"} (for non-yet-standard @*SHA3@-256/384/512) - with the addition of the {\f1\fs20 "none"} weak algo, to be used with caution;
- Computes and validates all JWT {\i claims}: dates, audiences, JWT ID;
- Thread-safe and high performance (2 us for a {\f1\fs20 HS256} verification under x64), with optional in-memory cache if needed (e.g. for slower {\f1\fs20 ES256});
- Stand-alone and cross-platform code: no external {\f1\fs20 dll}, works with @*Delphi@ or @*FPC@;
- Enhanced security - it is by design immune from @https://auth0.com/blog/2015/03/31/critical-vulnerabilities-in-json-web-token-libraries
- Full integration with the framework.
It is architectured around a set of classes, one per algorithm, following the least astonishment principle, and enhancing security:
\graph HierTJWTNone TJWTAbstract classes hierarchy
rankdir=LR;
\TJWTES256\TJWTAbstract
\TJWTSynSignerAbstract\TJWTAbstract
\TJWTHS256\TJWTSynSignerAbstract
\TJWTHS384\TJWTSynSignerAbstract
\TJWTHS512\TJWTSynSignerAbstract
\TJWTS3224\TJWTSynSignerAbstract
\TJWTS3256\TJWTSynSignerAbstract
\TJWTS3384\TJWTSynSignerAbstract
\TJWTS3512\TJWTSynSignerAbstract
\TJWTNone\TJWTAbstract
\
In {\f1\fs20 SynCrypto.pas} and {\f1\fs20 SynEcc.pas}, you will find:
- {\f1\fs20 TJWTAbstract} as abstract parent class for implementing JSON Web Tokens;
- {\f1\fs20 TJWTNone} implementing the {\f1\fs20 "none"} algorithm;
- {\f1\fs20 TJWTHS256 TJWTHS384 TJWTHS512} implementing the {\f1\fs20 "HS256 HS384 HS512"} algorithms, i.e. HMAC-SHA2 over 256, 384 or 512 bits;
- {\f1\fs20 TJWTS3256 TJWTS3384 TJWTS3512} implementing the {\f1\fs20 "S3256 S3384 S3512"} algorithms, i.e. SHA3 over 256, 384 or 512 bits;
- {\f1\fs20 TJWTES256} implementing the {\f1\fs20 "ES256"} algorithm, i.e. ECDSA using the P-256 curve and the SHA-256 hash algorithm.
To work with JWT, you may write for instance:
!var j: TJWTAbstract;
!    jwt: TJWTContent;
!...
!j := TJWTHS256.Create('secret',0,[jrcSubject],[]);
!try
!  j.Verify('eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibm'+
!    'FtZSI6IkpvaG4gRG9lIiwiYWRtaW4iOnRydWV9.TJVA95OrM7E2cBab30RMHrHDcEfxjoYZgeF'+
!    'ONFh7HgQ',jwt); // reference from jwt.io
!  check(jwt.result=jwtValid);
!  check(jwt.reg[jrcSubject]='1234567890');
!  check(jwt.data.U['name']='John Doe');
!  check(jwt.data.B['admin']);
!finally
!  j.Free;
!end;
The {\f1\fs20 'eyJhbGciOiJIUzI1NiIsIn...'} token contains in fact the following, once {\i base-64} decoded:
- header: {\f1\fs20 \{"alg":"HS256","typ":"JWT"\}}
- payload: {\f1\fs20 \{"sub":"1234567890","name":"John Doe","admin":true\}}
- signature: {\f1\fs20 HMACSHA256(base64UrlEncode(header) + "." + base64UrlEncode(payload), "secret")}
The {\f1\fs20 TJWTAbstract} classes implement the logic from supplied security parameters about a given set of JWT, then you can use {\f1\fs20 TJWTAbstract.Verify} to decode and check the payload and signature of a {\f1\fs20 JWT} into a {\f1\fs20 TJWTContent} local variable. As you can see, {\f1\fs20 TJWTContent.result} contains the decoding status, {\f1\fs20 TJWTContent.reg[]} the decoded claims, and {\f1\fs20 TJWTContent.data} is a @80@ giving access to any stored private information.
It has a built-in support of JWT {\i claims} when tokens are generated, so you can write:
!j := TJWTHS256.Create('sec',10,[jrcIssuer,jrcExpirationTime,jrcIssuedAt,jrcJWTID],[],60);
!token := j.Compute(['http://example.com/is_root',true],'joe');
Now, the {\f1\fs20 token} variable contains e.g. as signed payload:
${"http://example.com/is_root":true,"iss":"joe","iat":1482177879,"exp":1482181479,"jti":"1496DCE0676925DD33BB5A81"}
The issuer has been encoded as an expected {\f1\fs20 "iss":} field, {\f1\fs20 "iat"} and {\f1\fs20 "exp"} fields contain the issuing and expiration timestamps, and {\f1\fs20 "jti"} has been filled with an obfuscated {\f1\fs20 TSynUniqueIdentifier} as JWT ID. Since we use a {\f1\fs20 TJWTHS256} class, {\f1\fs20 @*HMAC-SHA256@} digital signature of the header and payload has then been appended - with a secret safely derivated from '{\f1\fs20 sec'} passphrase using 10 rounds of a {\f1\fs20 @*PBKDF2_HMAC_SHA256@} derivation (in practice, you may use a much higher number like 20,000).
Then you can decode such a token, and access its payload in a single method:
!j.Verify(token,jwt);
!assert(jwt.result=jwtValid);
!assert(jwt.reg[jrcIssuer]='joe');
Integration with method-based services is easy, using {\f1\fs20 TSQLRestServerURIContext.AuthenticationCheck} method:
!TMyDaemon = class(...
!protected
!  fJWT: TJWTAbstract;
!  ....
!procedure TMyDaemon.Files(Ctxt: TSQLRestServerURIContext);
!begin
!  if Ctxt.AuthenticationCheck(fJWT)=jwtValid then
!    Ctxt.ReturnFileFromFolder('c:\datafolder');
!end;
The above method will define a method-based service returning the content of a local folder, only if a valid {\f1\fs20 JWT} is supplied within the HTTP headers of the incoming request. If {\f1\fs20 AuthenticationCheck} fails to validate the token supplied in the associated {\f1\fs20 Ctxt}, if will return {\f1\fs20 401 HTTP_UNAUTHORIZED} to the client, as expected.
An alternative to use {\f1\fs20 JWT} for authentication may be to assign a {\f1\fs20 TJWTAbstract} inherited instance to {\f1\fs20 TSQLRestServer.JWTForUnauthenticatedRequest} - see @198@.
\page
: Handling errors
When using {\f1\fs20 Ctxt.Input*[]} properties, any missing parameter will raise an {\f1\fs20 EParsingException}. It will therefore be intercepted by the server process (as any other exception), and returned to the client with an error message containing the {\f1\fs20 Exception} class name and its associated message.
But you can have full access to the error workflow, if needed. In fact, calling either {\f1\fs20 Ctxt.Results()}, {\f1\fs20 Ctxt.Returns()}, {\f1\fs20 Ctxt.Success()} or {\f1\fs20 Ctxt.Error()} will specify the HTTP status code (e.g. 200 / "OK" for {\f1\fs20 Results()} and {\f1\fs20 Success()} methods by default, or 400 / "Bad Request" for {\f1\fs20 Error()}) as an {\f1\fs20 integer} value. For instance, here is how a service not returning any content can handle those status/error codes:
!procedure TSQLRestServer.Batch(Ctxt: TSQLRestServerURIContext);
!begin
!  if (Ctxt.Method=mPUT) and RunBatch(nil,nil,Ctxt) then
!    Ctxt.Success else
!    Ctxt.Error;
!end;
In case of an error on the server side, you may call {\f1\fs20 Ctxt.Error()} method (only the two valid status codes are {\f1\fs20 200} and {\f1\fs20 201}).
The {\f1\fs20 Ctxt.Error()} method has an optional parameter to specify a custom error message in plain English, which will be returned to the client in case of an invalid status code. If no custom text is specified, the framework will return the corresponding generic HTTP status text (e.g. {\f1\fs20 "Bad Request"} for default status code {\f1\fs20 HTTP_BADREQUEST} = 400).
In this case, the client will receive a corresponding serialized JSON error object, e.g. for {\f1\fs20 Ctxt.Error('Missing Parameter',HTTP_NOTFOUND)}:
${
$ "ErrorCode":404,
$ "ErrorText":"Missing Parameter"
$}
If called from an AJAX client, or a browser, this content should be easy to interpret.
Note that the framework core will catch any exception during the method execution, and will return a {\f1\fs20 "Internal Server Error" / HTTP_SERVERERROR} = 500 error code with the associated textual exception details.
\page
: Benefits and limitations of this implementation
Method-based services allow fast and direct access to all {\f1\fs20 mORMot} Client-Server {\f1\fs20 RESTful} features, over all usual protocols of our framework: @*HTTP@/1.1, Named Pipe, Windows Messages, direct in-memory/in-process access.
The {\i mORMot} implementation of method-based services gives full access to the lowest-level of the framework core, so it has some advantages:
- It can be tuned to fit any purpose (such as retrieving or returning some HTML or binary data, or modifying the HTTP headers on the fly);
- It is integrated into the @*REST@ful URI model, so it can be related to any table/class of our @*ORM@ framework (like {\f1\fs20 DataAsHex} service above), or it can handle any remote query (e.g. any @*AJAX@ or @*SOAP@ requests);
- It has a very low performance overhead, so can be used to reduce server workload for some common tasks.
Note that due to this implementation pattern, the {\i mORMot} service implementation is very fast, and not sensitive to the "Hash collision attack" security issue, as reported with {\i Apache} - see @http://blog.synopse.info/post/2011/12/30/Hash-collision-attack for details.
But with this implementation, a lot of process (e.g. parameter marshalling) is to be done by hand on both client and server side code. In addition, building and maintaining a huge SOA system with a "method by method" approach could be difficult, since it publishes one big "flat" set of services. This is were {\f1\fs20 interface}s enter the scene.
:46Interfaces
%cartoon05.png
: Delphi and interfaces
:  Declaring an interface
No, interface(-book) is not another social network, sorry.
In {\i Delphi} @*OOP@ model, an {\f1\fs20 @**interface@} defines a type that comprises abstract virtual methods. The short, easy definition is that an interface is a declaration of functionality without an implementation of that functionality. It defines "what" is available, not "how" it is made available. This is the so called "abstraction" benefit of interfaces (there are another benefits, like orthogonality of interfaces to classes, but we'll see it later).
In {\i Delphi}, we can declare an interface like so:
!type
!  ICalculator = interface(IInvokable)
!    ['{9A60C8ED-CEB2-4E09-87D4-4A16F496E5FE}']
!    /// add two signed 32-bit integers
!    function Add(n1,n2: integer): integer;
!  end;
It just sounds like a class definition, but, as you can see:
- It is named {\f1\fs20 ICalculator}, and not {\f1\fs20 TCalculator}: it is a common convention to start an interface name with a {\f1\fs20 I}, to make a difference with a {\f1\fs20 T} for a class or other implementation-level type definition;
- There is no visibility attribute (no {\f1\fs20 private / protected / public / published} keywords): in fact, it is just as if all methods were published;
- There is no fields, just methods (fields are part of the implementation, not of the interface): in fact, you can have properties in your interface definition, but those properties shall redirect to existing getter and setter methods, via {\f1\fs20 read} and {\f1\fs20 write} keywords;
- There is a strange number below the interface name, called a {\f1\fs20 @*GUID@}: this is an unique identifier of the interface - you can create such a genuine constant on the editor cursor position by pressing {\f1\fs20 Ctrl + Shift + G} in the {\i Delphi} IDE;
- But the methods are just defined as usual.
:  Implementing an interface with a class
Now that we have an interface, we need to create an implementation.
Our interface is very basic, so we may implement it like this:
!type
!!  TServiceCalculator = class(TInterfacedObject, ICalculator)
!  protected
!    fBulk: string;
!  public
!!    function Add(n1,n2: integer): integer;
!    procedure SetBulk(const aValue: string);
!  end;
!
!function TServiceCalculator.Add(n1, n2: integer): integer;
!begin
!  result := n1+n2;
!end;
!
!procedure TServiceCalculator.SetBulk(const aValue: string);
!begin
!  fBulk := aValue;
!end;
You can note the following:
- We added {\f1\fs20 ICalculator} name to the {\f1\fs20 class()} definition: this class inherits from {\f1\fs20 TInterfacedObject}, and implements the {\f1\fs20 ICalculator} interface;
- Here we have {\f1\fs20 protected} and {\f1\fs20 public} keywords - but the {\f1\fs20 Add} method can have any visibility, from the interface point of view: it will be used as implementation of an interface, even if the method is declared as {\f1\fs20 private} in the implementation class;
- There is a {\f1\fs20 SetBulk} method which is not part of the {\f1\fs20 ICalculator} definition - so we can add other methods to the implementation class, and we can even implement several interfaces within the same method (just add other interface names after like {\f1\fs20 class(TInterfacedObject, ICalculator, IAnotherInterface)};
- There a {\f1\fs20 fBulk} protected field member within this class definition, which is not used either, but could be used for the class implementation.
- Here we have to code an implementation for the {\f1\fs20 TServiceCalculator.Add()} method (otherwise the compiler will complain for a missing method), whereas there is no implementation expected for the {\f1\fs20 ICalculator.Add} method - it is perfectly "abstract".
:  Using an interface
Now we have two ways of using our {\f1\fs20 TServiceCalculator} class:
- The classic way;
- The abstract way (using an interface).
The "classic" way, using an explicit class instance:
!function MyAdd(a,b: integer): integer;
!var Calculator: TServiceCalculator;
!begin
!  Calculator := TServiceCalculator.Create;
!  try
!    result := Calculator.Add(a,b);
!  finally
!    Calculator.Free;
!  end;
!end;
Note that we used a {\f1\fs20 try..finally} block to protect the instance memory resource.
Then we can use an interface:
!function MyAdd(a,b: integer): integer;
!var Calculator: ICalculator;
!begin
!  Calculator := TServiceCalculator.Create;
!  result := Calculator.Add(a,b);
!end;
What's up over there?
- We defined the local variable as {\f1\fs20 ICalculator}: so it will be an {\f1\fs20 interface}, not a regular class instance;
- We assigned a {\f1\fs20 TServiceCalculator} instance to this {\f1\fs20 interface} variable: the variable will now handle the instance life time;
- We called the method just as usual - in fact, the computation is performed with the same exact expression: {\f1\fs20 result := Calculator.Add(a,b)};
- We do not need any {\f1\fs20 try...finally} block here: in {\i Delphi}, interface variables are {\i @*reference-counted@}: that is, the use of the interface is tracked by the compiler and the implementing instance, once created, is automatically freed when the compiler realizes that the number of references to a given interface variable is zero;
- And the performance cost is negligible: this is more or less the same as calling a virtual method (just one more redirection level).
In fact, the compiler creates an hidden {\f1\fs20 try...finally} block in the {\f1\fs20 MyAdd} function, and the instance will be released as soon as the {\f1\fs20 Calculator} variable is out of scope. The generated code could look like this:
!function MyAdd(a,b: integer): integer;
!var Calculator: TServiceCalculator;
!begin
!  Calculator := TServiceCalculator.Create;
!  try
!    Calculator.FRefCount := 1;
!    result := Calculator.Add(a,b);
!  finally
!    dec(Calculator.FRefCount);
!    if Calculator.FRefCount=0 then
!      Calculator.Free;
!  end;
!end;
Of course, this is a bit more optimized than this (and thread-safe), but you have got the idea.
:  There is more than one way to do it
One benefit of interfaces we have already told about, is that it is "orthogonal" to the implementation.
In fact, we can create another implementation class, and use the same interface:
!type
!!  TOtherServiceCalculator = class(TInterfacedObject, ICalculator)
!  protected
!!    function Add(n1,n2: integer): integer;
!  end;
!
!function TOtherServiceCalculator.Add(n1, n2: integer): integer;
!begin
!  result := n2+n1;
!end;
Here the computation is not the same: we use {\f1\fs20 n2+n1} instead of {\f1\fs20 n1+n2}... of course, this will result into the same value, but we can use this another method for our very same interface, by using its {\f1\fs20 TOtherServiceCalculator} class name:
!function MyOtherAdd(a,b: integer): integer;
!var Calculator: ICalculator;
!begin
!!  Calculator := TOtherServiceCalculator.Create;
!  result := Calculator.Add(a,b);
!end;
:  Here comes the magic
Now you may begin to see the point of using interfaces in a client-server framework like ours.
Our {\i mORMot} is able to use the same {\f1\fs20 interface} definition on both client and server side, calling all expected methods on both sides, but having all the implementation logic on the server side. The client application will transmit method calls (using JSON instead of much more complicated XML/@*SOAP@) to the server (using a "fake" implementation class created on the fly by the framework), then the execution will take place on the server (with obvious benefits), and the result will be sent back to the client, as JSON. The same interface can be used on the server side, and in this case, execution will be in-place, so very fast.
By creating a whole bunch of interfaces for implementing the business logic of your project, you will benefit of an open and powerful implementation pattern.
More on this later on... first we'll take a look at good principles of playing with interfaces.
\page
:47 SOLID design principles
The acronym @**SOLID@ is derived from the following @*OOP@ principles (quoted from the corresponding {\i Wikipedia} article):
- {\i Single responsibility principle}: the notion that an object should have only a single responsibility;
- {\i Open/closed principle}: the notion that "software entities ... should be open for extension, but closed for modification";
- {\i @Liskov substitution principle@}: the notion that "objects in a program should be replaceable with instances of their subtypes without altering the correctness of that program” - also named as "{\i design by contract}";
- {\i @*Interface@ segregation principle}: the notion that "many client specific interfaces are better than one general purpose interface.";
- {\i @*Dependency inversion@ principle}: the notion that one should "Depend upon Abstractions. Do not depend upon concretions.". {\i @*Dependency injection@} is one method of following this principle, which is also called {\i @*Inversion Of Control@} (aka @*IoC@).
If you have some programming skills, those principles are general statements you may already found out by yourself. If you start doing serious object-oriented coding, those principles are best-practice guidelines you will definitively gain following.
They certainly help to fight the three main code weaknesses:
- {\i Rigidity}: Hard to change something because every change affects too many other parts of the system;
- {\i Fragility}: When you make a change, unexpected parts of the system break;
- {\i Immobility}: Hard to reuse in another application because it cannot be disentangled from the current application.
:182  Single Responsibility Principle
When you define a class, it shall be designed to implement only one feature. The so-called feature can be seen as an "{\i axis of change}" or a "{\i a reason for change}".
Therefore:
- One class shall have only one reason that justifies changing its implementation;
- Classes shall have few dependencies on other classes;
- Classes shall be abstract from the particular layer they are running - see @7@.
For instance, a {\f1\fs20 TRectangle} object should not have both {\f1\fs20 ComputeArea} and {\f1\fs20 Draw} methods defined at once - they will define two responsibilities or axis of change: the first responsibility is to provide a mathematical model of a rectangle, and the second is to render it on GUI.
:   Splitting classes
To take an example from real coding, imagine you define a communication component. You want to communicate, say, with a bar-code scanner peripheral. You may define a single class, e.g. {\f1\fs20 TBarcodeScanner}, supporting such device connected over a serial port. Later on, the manufacturer deprecates the serial port support, since no computer still have it, and offer only USB models in its catalog. You may inherit from {\f1\fs20 TBarcodeScanner}, and add USB support.
\graph SOLIDSRP1 SOLID Principles - Single Responsibility: Single-to-rule-them-all class
\TBarcodeScanner\TUsbBarCodeScanner
\
But in practice, this new {\f1\fs20 TUsbBarCodeScanner} class is difficult to maintain, since it will inherit from serial-related communication.\line So you start splitting the class hierarchy, using an {\i abstract} parent class:
\graph SOLIDSRP2 SOLID Principles - Single Responsibility: Abstract parent class
\TAbstractBarcodeScanner\TSerialBarCodeScanner
\TAbstractBarcodeScanner\TUsbBarCodeScanner
\
We may define some {\f1\fs20 virtual abstract} methods, which will be overridden in inherited classes:
!type
!  TAbstractBarcodeScanner = class(TComponent)
!  protected
!    function ReadChar: byte; virtual; abstract;
!    function ReadFrame: TProtocolFrame; virtual; abstract;
!    procedure WriteChar(aChar: byte); virtual; abstract;
!    procedure WriteFrame(const aFrame: TProtocolFrame); virtual; abstract;
!    ...
Then, {\f1\fs20 TSerialBarCodeScanner} and {\f1\fs20 TUsbBarCodeScanner} classes will override those classes, according to the final implementation.
In fact, this approach is cleaner. But it is not perfect either, since it may be hard to maintain and extend. Imagine the manufacturer is using a standard protocol for communication, whatever USB or Serial connection is used. You will put this communication protocol (e.g. its state machine, its stream computation, its delaying settings) in the {\f1\fs20 TAbstractBarcodeScanner} class. But perhaps they will be diverse flavors, in {\f1\fs20 TSerialBarCodeScanner} or {\f1\fs20 TUsbBarCodeScanner}, or even due to diverse models and features (e.g. if it supports 2D or 3D bar-codes).
It appears that putting everything in a single class is not a good idea. Splitting protocol and communication appears to be preferred. Each "{\i axis of change}" - i.e. every aspect which may need modifications - requires its own class. Then the {\f1\fs20 T*BarcodeScanner} classes will {\i compose} protocols and communication classes within a single component.
Imagine we have two identified protocols (named {\f1\fs20 BCP1} and {\f1\fs20 BCP2}), and two means of communication (serial and USB). So we will define the following classes:
\graph SOLIDSRP3 SOLID Principles - Single Responsibility: Spliting protocol and communication
\TAbstractBarcodeProtocol\TBCP1BarcodeProtocol
\TAbstractBarcodeProtocol\TBCP2BarcodeProtocol
\TAbstractBarcodeConnection\TSerialBarcodeConnection
\TAbstractBarcodeConnection\TUsbBarcodeConnection
\
Then, we may define our final classes and components as such:
!type
!  TAbstractBarcodeConnection = class
!  protected
!    function ReadChar: byte; override;
!    procedure WriteChar(aChar: byte); override;
!    ...
!  TAbstractBarcodeProtocol = class
!  protected
!    fConnection: TAbstractBarcodeConnection;
!    function ReadFrame: TProtocolFrame; override;
!    procedure WriteFrame(const aFrame: TProtocolFrame); override;
!    ...
!  TAbstractBarcodeScanner = class(TComponent)
!  protected
!    fProtocol: TAbstractBarcodeProtocol;
!    fConnection: AbstractBarcodeConnection;
!  ...
And each actual inherited {\f1\fs20 class} will initialize the protocol and connection according to the expected model:
!constructor TSerialBarCodeScanner.Create(const aComPort: string; aBitRate: integer);
!begin
!  fConnection := TSerialBarcodeConnection(aComPort,aBitRate);
!  fProtocol := TBCP1BarcodeProtocol.Create(fConnection);
!end;
Here, we inject the connection instance to the protocol, since the later may need to read or write some bytes on the wire, when needed.
Another example is how our database classes are defined in {\f1\fs20 @*SynDB@.pas} - see @27@:
- The {\i connection properties} feature is handled by {\f1\fs20 TSQLDBConnectionProperties} classes;
- The actual {\i living connection} feature is handled by {\f1\fs20 TSQLDBConnection} classes;
- And {\i database requests} feature is handled by {\f1\fs20 TSQLDBStatement} instances using dedicated {\f1\fs20 NewConnection} / {\f1\fs20 ThreadSafeConnection} / {\f1\fs20 NewStatement} methods.
Therefore, you may change how a database connection is defined (e.g. add a property to a {\f1\fs20 TSQLDBConnectionProperties} child), and you won't have to change the statement implementation itself.
:   Do not mix UI and logic
Another practical "{\i Single Responsibility Principle}" smell may appear in your uses clause.
If your data-only or peripheral-only unit starts like this:
!unit MyDataModel;
!
!uses
!!  Winapi.Windows,
!  mORMot,
!  ...
It will induce a dependency about the {\i Windows} Operating System, whereas your data will certainly benefit from being OS-agnostic. Our todays compiler (Delphi or FPC) targets several OS, so coupling our data to the actual {\i Windows} unit does show a bad design.
Similarly, you may add a dependency to the VCL, via a reference to the {\f1\fs20 Forms} unit.\line If your data-only or peripheral-only unit starts like the following, beware!
!unit MyDataModel;
!
!uses
!!  Winapi.Messages,
!!  Vcl.Forms,
!  mORMot,
!  ...
If you later want to use {\f1\fs20 @*FMX@}, or {\f1\fs20 LCL} (from @*Lazarus@) in your application, or want to use your {\f1\fs20 MyDataModel} unit on a pure server application without any GUI, hosted on Windows - or even better on Linux/BSD - you are stuck.
Note that if you are used to developed in RAD mode, the units generated by the IDE wizards come with some default references in the {\f1\fs20 uses} clause of the generated {\f1\fs20 .pas} file. So take care of not introducing any coupling to your own business code!
As a general rule, our ORM/SOA framework source code tries to avoid such dependencies. All OS-specificities are centralized in our {\f1\fs20 SynCommons.pas} unit, and there is no dependency to the VCL when it is not mandatory, e.g. in {\f1\fs20 mORMot.pas}.
Following the RAD approach, you may start from your UI, i.e. defining the needed classes in the unit where you visual form (may be VCL or FMX) is defined. Don't follow this tempting, but dangerous path!
Code like the following may be accepted for a small example (e.g. the one supplied in the {\f1\fs20 SQlite3\\Samples} sub-folder of our repository source code tree), but is to be absolutely avoided for any production ready {\i mORMot}-based application:
!interface
!
!uses
!  Winapi.Windows, Winapi.Messages, System.SysUtils, System.Variants, System.Classes, Vcl.Graphics,
!  Vcl.Controls, Vcl.Forms, Vcl.Dialogs, mORMot, mORMotSQLite3;
!
!type
!  TForm1 = class(TForm)
!    procedure FormCreate(Sender: TObject);
!  private
!!    fModel: TSQLModel;
!!    fDatabase: TSQLRestServerDB;
!  public
!  end;
!
! implementation
!
!procedure TForm1.FormCreate(Sender: TObject);
!begin
!!  fModel := TSQLModel.Create([TSQLMyOwnRecord],'root');
!!  fDatabase := TSQLRestServerDB.Create(fModel,ChangeFileExt(paramstr(0),'.db'));
!end;
In your actual project units, when you define an @*ORM@ or @*SOA@ {\f1\fs20 class}, never include GUI methods within. In fact, the fact that our {\f1\fs20 @*TSQLRecord@} class definitions are common to both Client and Server sides makes this principle mandatory. You should not have any GUI related method on the Server side, and the Client side could use the objects instances with several GUI implementations ({\i Delphi} Client, AJAX Client...).
Therefore, if you want to change the GUI, you won't have to recompile the {\f1\fs20 TSQLRecord} class and the associated database model. If you want to deploy your server on a {\i @*Linux@} box (using e.g. {\i @*CrossKylix@} or @*FPC@ as compiler), you could reuse your very same code, since you do not have reference to the VCL in your business code.
This {\i Single responsibility principle} may sound simple and easy to follow (even obvious), but in fact, it is one of the hardest principles to get right. Naturally, we tend to join responsibilities in our class definitions. Our framework architecture will enforce you, by its @*Client-Server@ nature and all its high-level methods involving {\f1\fs20 interface}, to follow this principle, but it is always up to the end coder to design properly his/her types.
:158  Open/Closed Principle
When you define a class or a unit, at the same time:
- They shall be {\i open for extension};
- But {\i closed for modification}.
It means that you may be able to extend your existing code, without breaking its initial behavior.\line Some other guidelines may be added, but you got the main idea.
Conformance to this open/closed principle is what yields the greatest benefit of @*OOP@, i.e.:
- Code re-usability;
- Code maintainability;
- Code extendibility.
Following this principle will make your code far away from a regular RAD style. But benefits will be huge.
:   Applied to our framework units
When designing our ORM/SOA set of units, we tried to follow this principle. In fact, you should not have to modify its implementation. You should define your own units and classes, without the need to {\i hack} the framework source code.
Even if {\i Open Source} paradigm allows you to modify the supplied code, this shall not be done unless you are either fixing a bug or adding a new common feature. This is in fact the purpose of our @https://synopse.info web site, and most of the framework enhancements have come from user requests.
The framework Open Source @*license@ - see @34@ - may encourage user contributions in order to fulfill the Open/closed design principle:
- Your application code extends the {\i Synopse mORMot Framework} by defining your own classes or event handlers - this is how it is {\i open for extension};
- The main framework units shall remain inviolate, and common to all users - this illustrates the {\i closed for modification} design.
As a beneficial side effect, this principle will ensure that your code will be ready to follow the framework updates (which are quite regular). When a new version of {\i mORMot} is available, you should be able to retrieve it for free from our web site, replace your files locally, then build a new enhanced version of your application, with the benefit of all included fixes and optimizations. Even the source code repository is available - at @https://synopse.info/fossil or from @https://github.com/synopse/mORMot - and allows you to follow the current step of evolvment of the framework.
In short, abstraction is the key to peace of mind. All your code shall not depend on a particular implementation.
:   Open/Closed in practice
In order to implement this principle, several conventions could be envisaged:
- You shall better define some abstract classes, then use specific overridden classes for each and every implementation: this is for instance how @*Client-Server@ classes were implemented - see @35@;
- All object members shall be declared {\f1\fs20 private} or {\f1\fs20 protected} - this is a good idea to use @17@ for defining server-side process, and/or make the {\f1\fs20 @*TSQLRecord@} published properties read-only and using some client-side {\f1\fs20 constructor} with parameters;
- No singleton nor global variable - {\i ever};
- RTTI is dangerous - that is, let our framework use RTTI functions for its own cooking, but do not use it in your code.
In our previous bar-code scanner class hierarchy, we will therefore define the
!type
!  TAbstractBarcodeScanner = class(TComponent)
!  protected
!    fProtocol: TAbstractBarcodeProtocol;
!    fConnection: AbstractBarcodeConnection;
!  ...
!  public
!!    property Protocol: TAbstractBarcodeProtocol read fProtocol;
!!    property Connection: AbstractBarcodeConnection read fConnection;
!  ...
In this code, the actual variables are stored as {\f1\fs20 protected} fields, with only getters (i.e. {\b {\f1\fs20 read}}) in the {\f1\fs20 public} section. There is no setter (i.e. {\b {\f1\fs20 write}}) attribute, which may allow to change the {\f1\fs20 fProtocol/fConnection} instances in user code. You can still access those fields (it is mandatory in your inherited constructors), but user code should not use it.
As stated above - see @%%SOLIDSRP3@ - having dedicated classes for defining protocol and connection will also help implementing the {\i open/closed} principle. You will be able to define a new class, combining its own protocol and connection class instances, so it will be {\i Open for extension}. But you will not change the behavior of a class, by inheriting it: since protocol and connection are uncoupled, and used via {\i composition} in a dedicated class, it will be {\i Closed for modification}.
Using the newest {\f1\fs20 sealed} directive for a class may ensure that your {\f1\fs20 class} definition will follow this principle. If the class method or property is {\f1\fs20 sealed}, you will not be able to change its behavior in its inherited types, even if you are tempted to.
:   No Singleton nor global variables
About the @**singleton@ pattern, you should better always avoid it in your code. In fact, a singleton was a C++ (and Java) hack invented to implement some kind of global variables, hidden behind a static class definition. They were historically introduced to support mixed mode of application-wide initialization (mainly allocate the {\f1\fs20 stdio} objects needed to manage the console), and were abused in business logic.
Once you use a singleton, or a global variable, you will miss most of the benefit of OOP. A typical use of singleton is to register some class instances globally for the application. You may see some framework - or some part of the RTL - which will allow such global registration. But it will eventually void most benefits of proper dependency injection - see @157@ - since you will not be able to have diverse resolution of the same class.
For instance, if your database properties, or your application configuration are stored within a singleton, or a global variable, you will certainly not be able to use several database at once, or convert your single-user application with its GUI into a modern multi-user AJAX application:
!var
!  DBServer: string = 'localhost';
!  DBPort: integer = 1426;
!
!  UITextColor: TColor = clNavy;
!  UITextSize: integer = 12;
Such global variables are a smell of a broken {\i Open/Closed Principle}, since your project will definitively won't be open for extension. Using a {\f1\fs20 static class} variable (as allowed in newer version of Delphi), is just another way of defining a global variable, just adding the named scope of the {\f1\fs20 class} type.
Even if you do not define some global variable in your code, you may couple your code from an existing global variable. For instance, defining some variables with your {\f1\fs20 TMainForm = class(TForm)} class defined in the IDE, then using its global {\f1\fs20 MainForm: TMainForm} variable, or the {\f1\fs20 Application.MainForm} property, in your code. You will start to feel not right, when the unit where your {\f1\fs20 TMainForm} is defined will start to appear in your business code {\f1\fs20 uses} clause... just another global variable in disguise!
In our framework, we tried to never use global registration, but for the cases where it has been found safe to be implemented, e.g. when RTTI is cached, or JSON serialization is customized for a given type. All those informations will be orthogonal to the proper classes using them, so you may find some global variables in the framework units, only when it is worth it. For instance, we split {\f1\fs20 TSQLRecord}'s information into a {\f1\fs20 TSQLRecordProperties} for the shared intangible RTTI values, and {\f1\fs20 TSQLModelRecordProperties} instances, one per {\f1\fs20 TSQLModel}, for all the {\f1\fs20 TSQLModel/TSQLRest} specific settings - see @163@.
:159  Liskov Substitution Principle
Even if her name is barely unmemorable, {\i Barbara Liskov} is a great computer scientist, we should better learn from. It is worth taking a look at her presentation at @https://www.youtube.com/watch?v=GDVAHA0oyJU
The "{\i @**Liskov substitution principle@}" states that, if {\f1\fs20 TChild} is a subtype of {\f1\fs20 TParent}, then objects of type {\f1\fs20 TParent} may be replaced with objects of type {\f1\fs20 TChild} (i.e., objects of type {\f1\fs20 TChild} may be substitutes for objects of type {\f1\fs20 TParent}) without altering any of the desirable properties of that program (correctness, task performed, etc.).
The example given by {\i Barbara Liskov} was about stacks and queues: even if both do share {\f1\fs20 Push} and {\f1\fs20 Pop} methods, they should not inherit from a single parent type, since the storage behavior of a stack is quite the contrary of a queue. In your program, if you start to replace a stack by a queue, you will meet strange behaviors, for sure. According to proper {\i top-bottom} design flow, both types should be uncoupled. You may implement a {\f1\fs20 TFastStack} class using an in-memory list for storage, or another {\f1\fs20 TPersistedStack} class using a remote SQL engine, but both will have to behave like a {\f1\fs20 TStack}, i.e. according to the last-in first-out (LIFO) principle. On the other hand, any class implementing a {\i queue}  type should follow the the first-in first-out (FIFO) order, whatever kind of storage is used.
In practical {\i Delphi} code, relying on abstractions may be implemented by two means:
- Using only {\f1\fs20 abstract} parent {\f1\fs20 class} variables when consuming objects;
- Using {\f1\fs20 interface} variable instead of {\f1\fs20 class} implementations.
Here, we do not use inheritance for sharing implementation code, but for defining an expected behavior. Sometimes, you may break the {\i Liskov Substitution} principle in implementation methods which will be coded just to gather some reusable pieces of code (the {\i inheritance for implementation} pattern), preparing some behavior which may be used only by some of the subtypes. Such "internal" virtual methods of a subtype may change the behavior of its inherited method, for the sake of efficiency and maintainability. But with this kind of implementation inheritance, which is closer to plumbing than designing, methods should be declared as {\f1\fs20 protected}, and not published as part of the type definition.\line By the way, this is exactly what {\f1\fs20 interface} type definitions have to offer. You can inherit from another interface, and this kind of polymorphism should strictly follow the {\i Liskov Substitution} principle. Whereas the {\f1\fs20 class} types, implementing the interfaces, may use some protected methods which may break the principle, for the sake of code efficiency.
In order to fulfill this principle, you should:
- Properly name (and comment) your {\f1\fs20 class} or {\f1\fs20 interface} definition: having {\f1\fs20 Push} and {\f1\fs20 Pop} methods may be not enough to define a contract, so in this case type inheritance will define the expected expectation - as a consequence, you should better stay away from "duck typing" patterns, and dynamic languages, but rely on strong typing;
- Use the "behavior" design pattern, when defining your objects hierarchy - for instance, if a square may be a rectangle, a {\f1\fs20 TSquare} object is definitively {\i not} a {\f1\fs20 TRectangle} object, since the behavior of a {\f1\fs20 TSquare} object is not consistent with the behavior of a {\f1\fs20 TRectangle} object (square width always equals its height, whereas it is not the case for most rectangles);
- Write your tests using abstract local variables (and this will allow test code reuse for all children classes);
- Follow the concept of {\i Design by Contract}, i.e. the Meyer's rule defined as "{\i when redefining a routine [in a derivative], you may only replace its precondition by a weaker one, and its postcondition by a stronger one}" - use of preconditions and postconditions also enforce testing model;
- Separate your classes hierarchy: typically, you may consider using separated object types for implementing persistence and object creation (this is the common separation between {\i @*Factory@} and {\i @*Repository@} patterns).
:   Use parent classes
Within our framework, it will signify that {\f1\fs20 TSQLRestServer} or {\f1\fs20 TSQLRestClient} instances can be substituted to a {\f1\fs20 TSQLRest} object. Most @*ORM@ methods, especially at {\f1\fs20 TSQLRecord} level, expect an abstract {\f1\fs20 TSQLRest} parameter to be supplied - see @164@.
For instance, you may write:
!var anyRest: TSQLRest;
!    ID: TID;
!    rec1,rec2: TSQLMyRecord;
!...
!  ID := anyRest.Add(rec1,true);
!  rec2 := TSQLMyRecord.Create(anyRest,ID);
!...
And you may set any kind of actual class instance to {\f1\fs20 anyRest}, either a local stored database engine, or a HTTP remote access:
!  anyRest := TSQLRestServerDB.Create(aModel,'mydatabase.db');
!  anyRest := TSQLHttpClient.Create('1.2.3.4','8888',aModel,false);
You may even found in the {\f1\fs20 dddInfraSettings.pas} unit a powerful {\f1\fs20 TRestSettings.NewRestInstance()} method which is able to instantiate the needed {\f1\fs20 TSQLRest} inherited class from a set of JSON settings, i.e. either a {\f1\fs20 TSQLHttpClient}, or a local {\f1\fs20 TSQLRestServerFullMemory}, or a {\f1\fs20 TSQLRestServerDB} - the later either with a local {\i @*SQlite3@} database, an external @*SQL@ engine, or an external @*NoSQL@/@*MongoDB@ database.
Your code shall refer to abstractions, not to implementations. By using only methods and properties available at classes parent level, your code won't need to change because of a specific implementation.
:   I'm your father, Luke
You should note that, in the {\i Liskov substitution principle} definition, "parent" and "child" are no absolute. Which actual {\f1\fs20 class} is considered as "parent" may depend on the context use.
Most of the time, the parent may be the highest class in the hierarchy. For instance, in the context of a GUI application, you may use the most abstract class to access the application data, may it be stored locally, or remotely accessed over HTTP.
But when you initialize the {\f1\fs20 class} instance of a local stored server, you may need to setup the actual data storage, e.g. the file name or the remote SQL/NoSQL settings. In this context, you will need to access the "child" properties, regardless of the "parent" abstract use which will take care later on in the GUI part of the application.
Furthermore, in the context of data replication, server side or client side will have diverse behavior. In fact, they may be used as master or slave database, so in this case, you may explicitly define server or client {\f1\fs20 class} in your code. This is what our ORM does for its master/slave replication - see @147@.
If we come back to our bar-code scanner sample, most of your GUI code may rely on {\f1\fs20 TAbstractBarcodeScanner} components. But in the context of the application options, you may define the internal properties of each "child" class - e.g. the serial or USB port name, so in this case, your new "parent" class may be either {\f1\fs20 TSerialBarCodeScanner} or {\f1\fs20 TUsbCodeScanner}, or even better the {\f1\fs20 TSerialBarcodeConnection} or {\f1\fs20 TUsbBarcodeConnection} properties, to fulfill {\i Single Responsibility principle}.
:   Don't check the type at runtime
Some patterns shall never appear in your code. Otherwise, code refactoring should be done as soon as possible, to let your project be maintainable in the future.
Statements like the following are to be avoided, in either the parents' or the childs' methods:
!procedure TAbstractBarcodeScanner.SomeMethod;
!begin
!  if self is TSerialBarcodeScanner then
!  begin
!  ....
!  end
!  else
!  if self is TUsbBarcodeScanner then
!...
Or, in its disguised variation, using an @*enumerated@ item:
!case fProtocol.MeanOfCommunication of
!meanSerial: begin
!  ...
!end;
!meantUsb:
!  ...
This later piece of code does not check {\f1\fs20 self}, but the {\f1\fs20 fProtocol} protected field. So even if you try to implement the {\i Single Responsibility principle}, you may still be able to break {\i Liskov Substitution}!
Note that both patterns will eventually break the {\i Single Responsibility principle}: each behavior shall be defined in its own child {\f1\fs20 class} methods. As the {\i Open/Closed principle} will also be broken, since the class won't be open for extension, without touching the parent class, and modify the nested {\f1\fs20 if self is T* then ...} or {\f1\fs20 case fProtocol.* of ...} expressions.
:   Partially abstract classes
Another code smell may appear when you define a method which will stay {\f1\fs20 abstract} for some children, instantiated in the project. It will imply that some of the parent {\f1\fs20 class} behavior is not implemented at this particular hierarchy level. So you will not be able to use all the parent's methods, as will be expected by the {\i Liskov Substitution principle}.\line Note that the compiler will complain for it, hinting that you are creating a class with abstract methods. Never ignore such hints - which may benefit for being handled as errors at compilation time. The (in)famous "{\f1\fs20 Abstract Error}" error dialog, which may appear at runtime, will reflect this bad code implementation. When it occurs on a server application without GUI... you got a picture of the terror, I guess...
A more subtle violation of {\i Liskov} may appear if you break the expectation of the parent class. The following code, which emulates a bar-code reader peripheral by sending the frame by email for debugging purpose (why not?), clearly fails the {\i Design by Contract} approach:
!  TEMailEmulatedBarcodeProtocol = class(TAbstractBarcodeProtocol)
!  protected
!    function ReadFrame: TProtocolFrame; override;
!    procedure WriteFrame(const aFrame: TProtocolFrame); override;
!    ...
!
!function TEMailEmulatedBarcodeProtocol.ReadFrame: TProtocolFrame;
!begin
!  raise EBarcodeException.CreateUTF8('%.ReadFrame is not implemented!',[self]);
!end;
!
!procedure TEMailEmulatedBarcodeProtocol.WriteFrame(const aFrame: TProtocolFrame);
!begin
!  SendEmail(fEmailNotificationAddress,aFrame.AsString);
!end;
We expected this class to fully implement the {\f1\fs20 TAbstractBarcodeProtocol} contract, whereas calling {\f1\fs20 TEMailEmulatedBarcodeProtocol.ReadFrame} will not be able to read any data frame, but will raise an exception. So we can not use this {\f1\fs20 TEMailEmulatedBarcodeProtocol class} as replacement to any other {\f1\fs20 TAbstractBarcodeProtocol} class, otherwise it will fail at runtime.\line A correct implementation may perhaps to define a {\f1\fs20 TFakeBarcodeProtocol} class, implementing all the parent methods via a set of events or some text-based scenario, so that it will behave just like a correct {\f1\fs20 TAbstractBarcodeProtocol} class, in the full extend of its expectations.
:   Messing units dependencies
Last but not least, if you need to explicitly add child classes units to the parent class unit {\f1\fs20 uses} clause, it looks like if you just broke the {\i Liskov Substitution principle}.
!unit AbstractBarcodeScanner;
!
!uses
!  SysUtils,
!  Classes,
!!  SerialBarcodeScanner; // Barbara complains: "it smells"!
!!  UsbBarcodeScanner;    // Barbara complains: "it smells"!
! ...
If your code is like this, you will have to remove the reference to the inherited classes, for sure.
Even a dependency to one of the low-level implementation detail is to be avoided:
!unit AbstractBarcodeScanner;
!
!uses
!!  Windows,
!  SysUtils,
!  Classes,
!!  ComPort;
! ...
Your abstract parent {\f1\fs20 class} should {\b not} be coupled to a particular {\i Operating System}, or a mean of communication, which may not be needed. Why will you add a dependency to raw {\i RS-232} communication protocol, which is very likely to be deprecated?
One way of getting rid of this dependency is to define some abstract types (e.g. enumerations or simple structures like {\f1\fs20 record}), which will then be translated into the final types as expected by the {\f1\fs20 ComPort.pas} or {\f1\fs20 Windows.pas} units. Consider putting all the child classes dependencies at {\f1\fs20 constructor} level, and/or use {\f1\fs20 class} composition via the {\i Single Responsibility principle} so that the parent {\f1\fs20 class} definition will not be polluted by implementation details of its children.
You my also use a {\i registration list}, maintained by the parent unit, which may be able to register the classes implementing a particular behavior at runtime. Thanks to {\i Liskov}, you will be able to {\i substitute} any parent class by any of its inherited implementation, so defining the types at runtime only should not be an issue.
:   Practical advantages
The main advantages of this coding pattern are the following:
- Thanks to this principle, you will be able to {\i @*stub@} or {\i @*mock@} an @*interface@ or a {\f1\fs20 class} - see @62@ - e.g. uncouple your object persistence to the actual database it runs on: this principle is therefore mandatory for implementing unitary @*test@ing to your project;
- Furthermore, testing will be available not only at isolation level (testing each child class), but also at abstracted level, i.e. from the client point of view - you can have implementation which behave correctly when tested individually, but which failed when tested at higher level if the {\i Liskov} principle was broken;
- As we have seen, if this principle is violated, the other principles are very likely to be also broken - e.g. the parent class will need to be modified whenever a new derivative of the base class is defined (violation of the {\i Open/Closed} principle), or your {\f1\fs20 class} types may implement more than one behavior at a time (violation of the {\i Single Responsibility} principle);
- Code re-usability is enhanced by method re-usability: a method defined at a parent level does not require to be implemented for each child.
The @*SOA@ and @*ORM@ concepts, as implemented by our framework, try to be compliant with the {\i Liskov substitution principle}. It is true at {\f1\fs20 class} level for the ORM, but a more direct {\i Design by Contract} implementation pattern is also available, since the whole SOA stack involves a wider usage of {\f1\fs20 @*interface@s} in your projects.
:160  Interface Segregation Principle
This principle states that once an @*interface@ has become too 'fat' it shall be split into smaller and more specific interfaces so that any clients of the interface will only know about the methods that pertain to them. In a nutshell, no client should be forced to depend on methods it does not use.
As a result, it will help a system stay decoupled  and thus easier to re-factor, change, and redeploy.
:   Consequence of the other principles
{\i Interface segregation} should first appear at {\f1\fs20 class} level. Following the {\i Single Responsibility} principle, you are very likely to define several smaller classes, with a small extent of methods. Then use dedicated types of class, relying on composition to expose its own higher level set of methods.
The bar-code class hierarchy illustrates this concept. Each {\f1\fs20 T*BarcodeProtocol} and {\f1\fs20 T*BarcodeConnection} class will have its own set of methods, dedicated either to protocol handling, or data transmission. Then the {\f1\fs20 T*BarCodeScanner} classes will {\i compose} those smaller classes into a new class, with a single event handler:
!type
!  TOnBarcodeScanned = procedure(Sender: TAbstractBarcodeScanner; const Barcode: string) of object;
!
!  TAbstractBarcodeScanner = class(TComponent)
!  ...
!!  property OnBarcodeScanned: TOnBarcodeScanned read fOnBarcodeScanned write fOnBarcodeScanned;
!  ...
This single {\f1\fs20 OnBarcodeScanned} event will be the published property of the component. Both protocol and connection details will be hidden within the internal classes. The final application will use this event, and react as expected, without actually knowing anything about the implementation details.
:   Using interfaces
The @*SOA@ part of the framework allows direct use of {\f1\fs20 interface} types to implement services. This great @*Client-Server@ @*SOA@ implementation pattern - see @11@ - helps decoupling all services to individual small methods. In this case also, the @*stateless@ used design will also reduce the use of 'fat' session-related processes: an object life time can be safely driven by the {\f1\fs20 interface} scope.
By defining {\i Delphi} {\f1\fs20 interface} instead of plain {\f1\fs20 class}, it helps creating small and business-specific contracts, which can be executed on both client and server side, with the same exact code.
Since the framework makes interface consumption and publication very easy, you won't be afraid of exposing your implementation classes as small pertinent interface.\line For instance, if you want to publish a third-party API, you may consider publishing dedicated interfaces, each depending on every API consumer expectations. So your main implementation logic won't be polluted by how the API is consumed, and, as correlative, the published API may be closer to each particular client needs, without been polluted by the other client needs. @*DDD@ will definitively benefit for {\i Interface Segregation}, since this principle is the golden path to avoid {\i domain leaking} - see @156@.
:157  Dependency Inversion Principle
Another form of decoupling is to invert the dependency between high and low level of a software design:
- High-level modules should not depend on low-level modules. Both should depend on abstractions;
- Abstractions should not depend upon details. Details should depend upon abstractions.
The goal of the {\i @**dependency inversion@ principle} is to decouple high-level components from low-level components such that reuse with different low-level component implementations becomes possible. A simple implementation pattern could be to use only @*interface@s owned by, and existing only with the high-level component package.
This principle results in {\i @**Inversion Of Control@} (aka @**IoC@): since you rely on the abstractions, and try not to depend upon concretions (i.e. on implementation details), you should first concern by defining your interfaces.
:   Upside Down Development
In conventional application architecture, lower-level components are designed to be consumed by higher-level components which enable increasingly complex systems to be built. This design limits the reuse opportunities of the higher-level components, and certainly breaks the {\i @Liskov substitution principle@}.
For our bar-code reader sample, we may be tempted to start from the final {\f1\fs20 TSerialBarcodeScanner} we need in our application. We were asked by our project leader to allow bar-code scanning in our flagship application, and the extend of the development has been reduced to support a single model of device, in RS-232 mode - this may be the device already owned by our end customer.
This particular customer may have found some RS-232 bar-code relics from the 90s in its closets, but, as an experience programmer, you know that the next step will be to support USB, in a very close future. All this bar-code reading stuff will be marketized by your company, so it is very likely that another customer will very soon ask for using its own brand new bar-code scanners... which will support only USB.
So you will modelize your classes as with @%%SOLIDSRP2@ and @%%SOLIDSRP3@. Even if the {\f1\fs20 TUsbBarCodeScanner} - and its correlative {\f1\fs20 TUsbBarcodeConnection class} - is not written, nor tested (you do not even have an actual USB bar-code scanner to do proper testing yet!), you are prepared for it.
When you will eventually add USB support, the UI part of the application won't have to be touched. Just implementing your new inherited class, leveraging all previous coding. Following {\i Dependency Inversion} from the beginning will definitively save your time. Even in an {\i Agile} kind of process - where "{\i Responding to change}" is most valuable - the small amount of work on implementing first from the abstraction with the initial implementation will be very beneficial.
In fact, this {\i Dependency Inversion} principle is a prerequisite for proper {\i @**Test-Driven@} Design. Following this @**TDD@ pattern, you first write your test, then fail your test, then write the implementation. In order to write the test, you need the abstracted interface of the feature to be available. So you will start from the abstraction, then write the concretion.
:   Injection patterns
In other languages (like Java or .Net), various patterns such as {\i Plug-in, Service Locator}, or {\i @*Dependency Injection@} are then employed to facilitate the run-time provisioning of the chosen low-level component implementation to the high-level component.
Our @*Client-Server@ architecture facilitates this decoupling pattern for its @*ORM@ part, and allows the use of native {\i Delphi} {\f1\fs20 interface} to call services from an abstract @*factory@, for its @*SOA@ part.
A set of dedicated classes, defined in {\f1\fs20 mORMot.pas}, allows to leverage {\f1\fs20 @*IoC@}: see e.g. {\f1\fs20 TInjectableObject}, {\f1\fs20 TInterfaceResolver}, {\f1\fs20 TInterfaceResolverForSingleInterface} and {\f1\fs20 TInterfaceResolverInjected}, which may be used in conjunction with {\f1\fs20 TInterfaceStub} or {\f1\fs20 TServiceContainer} high-level mocking and SOA features of the framework - see @62@ and @63@.
: Circular reference and (zeroing) weak pointers
:  Weak pointers
The memory allocation model of the {\i Delphi} {\f1\fs20 interface} type uses some kind of {\i Automatic Reference Counting} (@*ARC@). In order to avoid memory and resource leaks and potential random errors in the applications (aka the terrible {\f1\fs20 EAccessViolation} exception on customer side) when using @46@, a @*SOA@ framework like {\i mORMot} has to offer so-called {\i @**Weak pointers@} and {\i @**Zeroing Weak pointers@} features.
By default in {\i Delphi}, all references are defined:
- as {\i weak references} for pointer and class instances;
- with {\i explicit copy} for low-level value types like {\f1\fs20 integer, Int64, @*currency@, @*double@} or {\f1\fs20 record} (and old deprecated {\f1\fs20 object} or {\f1\fs20 shortstring});
- via {\i copy-on-write} with {\i reference counting} for high-level value types (e.g. {\f1\fs20 string, @*widestring@, variant} or a {\i dynamic array} - with the exception of tuned memory handling for @80@);
- as {\i strong reference} with {\i reference counting} for {\f1\fs20 interface} instances.
The main issue with {\i strong reference counting} is the potential {\i circular reference} problem.\line This occurs when an {\f1\fs20 interface} has a strong pointer to another, but the target {\f1\fs20 interface} has a strong pointer back to the original. Even when all other references are removed, they still will hold on to one another and won't be released. This can also happen indirectly, by a chain of objects that might have the last one in the chain referring back to an earlier object.
See the following {\f1\fs20 interface} definition for instance:
!  IParent = interface
!    procedure SetChild(const Value: IChild);
!    function GetChild: IChild;
!    function HasChild: boolean;
!    property Child: IChild read GetChild write SetChild;
!  end;
!
!  IChild = interface
!    procedure SetParent(const Value: IParent);
!    function GetParent: IParent;
!    property Parent: IParent read GetParent write SetParent;
!  end;
The following implementation will definitively leak memory:
!procedure TParent.SetChild(const Value: IChild);
!begin
!  FChild := Value;
!end;
!
!procedure TChild.SetParent(const Value: IParent);
!begin
!  FParent := Value;
!end;
In {\i Delphi}, most common kind of reference-copy variables (i.e. {\f1\fs20 variant}, {\i dynamic array} or {\f1\fs20 string}) solve this issue by implementing {\i copy-on-write}. Unfortunately, this pattern is not applicable to {\f1\fs20 interface}, which are not {\i value} objects, but {\i reference} objects, tied to an implementation {\f1\fs20 class}, which can't be copied.
One common solution is to use {\i Weak pointers}, by which the {\f1\fs20 interface} is assigned to a property without incrementing the reference count.
Note that @*garbage collector@ based languages (like Java or C#) do not suffer from this problem, since the circular references are handled by their memory model: objects lifetime are maintained globally by the memory manager. Of course, it will increase memory use, slowdown the process due to additional actions during allocation and assignments (all objects and their references have to be maintained in internal lists), and may slow down the application when garbage collector enters in action. In order to avoid such issues when performance matters, experts tend to pre-allocate and re-use objects: this is one common limitation of this memory model, and why {\i Delphi} is still a good candidate (like unmanaged C or C++ - and also {\i Objective C}) when it deals with performance and stability. In some cases (e.g. when using an object cache), such languages have to introduce some kind of "weak pointers", to allow some referenced objects to be reclaimed by garbage collection: but it is a diverse mechanism, under the same naming.
:  Handling weak pointers
In order to easily create a weak pointer, the following function was added to {\f1\fs20 mORMot.pas}:
!procedure SetWeak(aInterfaceField: PIInterface; const aValue: IInterface);
!begin
!  PPointer(aInterfaceField)^ := Pointer(aValue);
!end;
It will assign the {\f1\fs20 interface} reference to a field by assigning the {\f1\fs20 pointer} of this instance to the internal field. It will by-pass the reference counting, so memory won't be leaked any more.
Therefore, it could be used as such:
!procedure TParent.SetChild(const Value: IChild);
!begin
!  SetWeak(@FChild,Value);
!end;
!
!procedure TChild.SetParent(const Value: IParent);
!begin
!  SetWeak(@FParent,Value);
!end;
:  Zeroing weak pointers
But there are still some cases where it is not enough. Under normal circumstances, a {\f1\fs20 class} instance should not be deallocated if there are still outstanding references to it. But since weak references don't contribute to an {\f1\fs20 interface} reference count, a {\f1\fs20 class} instance can be released when there are outstanding weak references to it. Some memory leak or even random access violations could occur. A debugging nightmare...
In order to solve this issue, ARC's {\i Zeroing Weak pointers} come to mind.\line It means that weak references will be set to {\f1\fs20 nil} when the object they reference is released. When this happens, the automatic zeroing of the outstanding weak references prevents them from becoming dangling pointers. And {\i voilà}! No access violation any more!
Such a {\i Zeroing} ARC model has been implemented in {\i Objective C} by Apple, starting with Mac OS X 10.7 Lion, in replacement (and/or addition) to the previous manual memory handling implementation pattern: in its Apple's flavor, ARC is available not only for interfaces, but for objects, and is certainly more sophisticated than the basic implementation available in the {\i Delphi} compiler: it is told (at least from the marketing paper point of view) to use some deep knowledge of the software architecture to provide an accurate access to all instances - whereas the {\i Delphi} compiler just relies on a {\i out-of-scope} pattern. In regard to classic {\i garbage collector} memory model, ARC is told to be much more efficient, due to its deterministic nature: Apple's experts ensure that it does make a difference, in term of memory use and program latency - which both are very sensitive on "modest" mobile devices. In short, thanks to ARC, your phone UI won't glitch during background garbage recycling. So {\f1\fs20 mORMot} will try to offer a similar feature, even if the {\i Delphi} compiler does not implement it (yet).
In order to easily create a so-called zeroing weak pointer, the following function was defined in {\f1\fs20 mORMot.pas}:
!procedure SetWeakZero(aObject: TObject; aObjectInterfaceField: PIInterface;
!  const aValue: IInterface);
A potential use case could be:
!procedure TParent.SetChild(const Value: IChild);
!begin
!  SetWeakZero(self,@FChild,Value);
!end;
!
!procedure TChild.SetParent(const Value: IParent);
!begin
!  SetWeakZero(self,@FParent,Value);
!end;
We also defined a {\f1\fs20 class helper} around the {\f1\fs20 TObject} class, to avoid the need of supplying the {\f1\fs20 self} parameter, but unfortunately, the {\f1\fs20 class helper} implementation is so buggy it won't be even able to compile before {\i Delphi} XE version of the compiler. But it will allow to write code as such:
!procedure TParent.SetChild(const Value: IChild);
!begin
!  SetWeak0(@FChild,Value);
!end;
For instance, the following code is supplied in the regression tests, and will ensure that weak pointers are effectively zeroed when {\f1\fs20 SetWeakZero()} is used:
!function TParent.HasChild: boolean;
!begin
!  result := FChild<>nil;
!end;
!
!  Child := nil; // here Child is destroyed
!  Check(Parent.HasChild=(aWeakRef=weakref),'ZEROed Weak');
Here, {\f1\fs20 aWeakRef=weakref} is {\f1\fs20 true} when {\f1\fs20 SetWeak()} has been called, and equals {\f1\fs20 false} when {\f1\fs20 SetWeakZero()} has been used to assign the {\f1\fs20 Child} element to its {\f1\fs20 Parent} interface.
:  Weak pointers functions implementation details
The {\f1\fs20 SetWeak()} function itself is very simple. The {\i Delphi} RTL/VCL itself use similar code when necessary.
But the {\f1\fs20 SetWeakZero()} function has a much more complex implementation, due to the fact that a list of all weak references has to be maintained per {\f1\fs20 class} instance, and set to {\f1\fs20 nil} when this referring instance is released.
The {\i mORMot} implementation tries to implement:
- Best performance possible when processing the {\i Zeroing} feature;
- No performance penalty for other classes not involved within weak references;
- Low memory use, and good scalability when references begin to define huge graphs;
- Thread safety - which is mandatory at least on the server side of our framework;
- Compatible with {\i Delphi} 6 and later (avoid syntax tricks like {\f1\fs20 generic}).
Some good existing implementations can be found on the Internet:
- {\i Andreas Hausladen} provided a classical and complete implementation at @http://andy.jgknet.de/blog/2009/06/weak-interface-references using some nice tricks (like per-instance optional speed up using a void {\f1\fs20 IWeakInterface interface} whose VMT slot will refer to the references list), is thread-safe and is compatible with most {\i Delphi} versions - but it will slow down all {\f1\fs20 TObject.FreeInstance} calls (i.e. within {\f1\fs20 Free / Destroy}) and won't allow any overridden {\f1\fs20 FreeInstance} method implementation;
- {\i Vincent Parrett} proposed at @http://www.finalbuilder.com/Resources/Blogs/PostId/410/WeakRefence-in-Delphi-solving-circular-interfac.aspx a {\f1\fs20 generic}-based solution (not thread-safe nor optimized for speed), but requiring to inherit from a base class for any {\f1\fs20 class} that can have a weak reference pointing to it;
- More recently, {\i Stefan Glienke} published at @http://delphisorcery.blogspot.fr/2012/06/weak-interface-references.html another {\f1\fs20 generic}-based solution, not requiring to inherit from a base class, but not thread-safe and suffering from the same limitations related to {\f1\fs20 TObject.FreeInstance}.
The implementation included within {\i mORMot} uses several genuine patterns, when compared to existing solutions:
- It will hack the {\f1\fs20 TObject.FreeInstance} at the {\f1\fs20 class} VMT level, so will only slow down the exact {\f1\fs20 class} which is used as a weak reference, and not others (also its inherited {\f1\fs20 classes} won't be overridden) - and it will allow custom override of the {\f1\fs20 virtual FreeInstance} method;
- It makes use of our {\f1\fs20 TDynArrayHashed} wrapper to provide a very fast lookup of instances and references, without using {\f1\fs20 generic} definitions - hashing will start when it will be worth it, i.e. for any list storing more than 32 items;
- The unused {\f1\fs20 vmtAutoTable} VMT slot is used to handle the class-specific orientation of this feature (similar to {\f1\fs20 TSQLRecordProperties} lookup as implemented for @DI-2.1.3@), for best speed and memory use.
See the {\f1\fs20 TSetWeakZeroClass} and {\f1\fs20 TSetWeakZeroInstance} implementation in {\f1\fs20 mORMot.pas} for the details.
\page
:62 Interfaces in practice: dependency injection, stubs and mocks
In order to fulfill the @47@, two features are to be available when handling interfaces:
- {\i @*Dependency injection@} or {\i @*Inversion of Control@} (aka @**IoC@) - see @157@;
- {\i @*Stub@bing} and {\i mocking} of interfaces for proper testing.
We will show now how {\i mORMot} provides all needed features for such patterns, testing a simple "forgot my password" scenario: a password shall be computed for a given user name, then transmitted via SMS, and its record shall be updated in the database.
:162  Dependency Injection at constructors
A direct implementation of dependency injection at a {\f1\fs20 class} level can be implemented in {\i Delphi} as such:
- All external dependencies shall be defined as abstract {\f1\fs20 interface};
- An external @*factory@ could be used to retrieve an {\f1\fs20 interface} instance, {\b or} class {\f1\fs20 constructor} shall receive the dependencies as parameters.
Using an external factory can be made within {\i mORMot} via {\f1\fs20 @**TServiceFactory@} - see @63@. Automated dependency injection is also available via a set of classes, uncoupled from the @*SOA@ features of the framework, mainly {\f1\fs20 TInjectableObject} and {\f1\fs20 TInterfaceResolver} types, and their inherited classes - see @161@.
Here, we will use the more direct {\f1\fs20 constructor}-based pattern for a simple "forgot my password" scenario.
This is the class we want to test:
!  TLoginController = class(TInterfacedObject,ILoginController)
!  protected
!    fUserRepository: IUserRepository;
!    fSmsSender: ISmsSender;
!  public
!    constructor Create(const aUserRepository: IUserRepository;
!      const aSmsSender: ISmsSender);
!    procedure ForgotMyPassword(const UserName: RawUTF8);
!  end;
The {\f1\fs20 constructor} will indeed {\i inject} its dependencies into its own instance:
!constructor TLoginController.Create(const aUserRepository: IUserRepository;
!  const aSmsSender: ISmsSender);
!begin
!  fUserRepository := aUserRepository;
!  fSmsSender := aSmsSender;
!end;
The dependencies are defined with the following two interfaces(only the needed methods are listed here, but a real interface may have much more members, but not too much, to follow the {\i interface segregation} @*SOLID@ principle):
!  IUserRepository = interface(IInvokable)
!    ['{B21E5B21-28F4-4874-8446-BD0B06DAA07F}']
!    function GetUserByName(const Name: RawUTF8): TUser;
!    procedure Save(const User: TUser);
!  end;
!  ISmsSender = interface(IInvokable)
!    ['{8F87CB56-5E2F-437E-B2E6-B3020835DC61}']
!    function Send(const Text, Number: RawUTF8): boolean;
!  end;
Note also that all those code will use a plain {\f1\fs20 record} as {\i @**Data Transfer Object@} (@*DTO@):
!  TUser = record
!    Name: RawUTF8;
!    Password: RawUTF8;
!    MobilePhoneNumber: RawUTF8;
!    ID: TID;
!  end;
Here, we won't use {\f1\fs20 TSQLRecord} nor any other {\f1\fs20 class}es, just plain {\f1\fs20 record}s, which will be used as neutral means of transmission. The difference between {\i Data Transfer Objects} and {\i business objects} or {\i @**Data Access Objects@} (@**DAO@) like our {\f1\fs20 TSQLRecord} is that a DTO does not have any behavior except for storage and retrieval of its own data. It can also be independent to the persistence layer, as implemented underneath our business domain. Using a {\f1\fs20 record} in {\i Delphi} ensure it won't be part of a complex business logic, but will remain used as value objects.
Now, let's come back to our {\f1\fs20 TLoginController} class.\line Here is the method we want to test:
!procedure TLoginController.ForgotMyPassword(const UserName: RawUTF8);
!var U: TUser;
!begin
!  U := fUserRepository.GetUserByName(UserName);
!  U.Password := Int32ToUtf8(Random(MaxInt));
!  if fSmsSender.Send('Your new password is '+U.Password,U.MobilePhoneNumber) then
!    fUserRepository.Save(U);
!end;
It will retrieve a {\f1\fs20 TUser} instance from its @*repository@, then compute a new password, and send it via SMS to the user's mobile phone. On success, it is supposed to persist (save) the new user information to the database.
:  Why use fake / emulated interfaces?
Using the real implementation of {\f1\fs20 IUserRepository} will expect a true database to be available, with some potential issues on existing data. Similarly, the class implementing {\f1\fs20 ISmsSender} in the final project should better not to be called during the test phase, since sending a SMS does cost money, and we will need a true mobile phone or Internet gateway to send the password.
For our testing purpose, we only want to ensure that when the "forgot my password" scenario is executed, the user record modification is persisted to the database.
One possibility could be to define two new dedicated {\f1\fs20 class}es, implementing both {\f1\fs20 IUserRepository} and {\f1\fs20 ISmsSender} interfaces. But it will be obviously time consuming and error-prone. This may be typical case when writing the test could be more complex than writing the method to be tested.
In order to maximize your ROI, and allow you to focus on your business logic, the {\i mORMot} framework proposes a simple and efficient way of creating "fake" implementations of any {\f1\fs20 interface}, just by defining the minimum behavior needed to run the test.
:166   Stubs and mocks
In the book "{\i The Art of Unit Testing}" (Osherove, Roy - 2009), a distinction is drawn between {\i @**stub@} and {\i @**mock@} objects:
- {\i Stubs} are the simpler of the two families of fake objects, simply implementing the same interface as the object that they represent and returning pre-arranged responses. Thus a fake object merely provides a set of method stubs. Therefore the name. In {\i mORMot}, it is created via the {\f1\fs20 TInterfaceStub} generator;
- {\i Mocks} are described as a fake object that helps decide if a test failed or passed, by verifying if an interaction on an object occurred or not. Everything else is defined as a stub. In {\i mORMot}, it is created via the {\f1\fs20 TInterfaceMock} generator, which will link the fake object to an existing {\f1\fs20 TSynTestCase} instance - see @12@.
In practice, there should be only {\b one} mock per test, with as many stubs as necessary to let the test pass. Using a mocking/stubbing framework allows quick on-the-fly generation of {\f1\fs20 interface} with unique behavior dedicated to a particular test. In short, you define the stubs needed to let your test pass, and define one mock which will pass or fail the test depending on the feature you want to test.
Our {\i mORmot} framework follows this distinction, by defining two dedicated classes, named {\f1\fs20 TInterfaceStub} and {\f1\fs20 TInterfaceMock}, able to define easily the behavior of such classes.
:   Defining stubs
Let's implement our "forgot my password" scenario test.
The {\f1\fs20 TSynTestCase} child method could start as such:
!procedure TMyTest.ForgetThePassword;
!var SmsSender: ISmsSender;
!    UserRepository: IUserRepository;
This is all we need: one dedicated test case method, and our two local variables, ready to be set with our stubbed / mocked implementation classes.
First of all, we will need to implement {\f1\fs20 ISmsSender.Send} method. We should ensure that it returns {\f1\fs20 true}, to indicate a successful sending.
With {\i mORMot}, it is as simple as:
!  TInterfaceStub.Create(TypeInfo(ISmsSender),SmsSender).
!    Returns('Send',[true]);
It will create a fake class (here called a "@*stub@") emulating the whole {\f1\fs20 ISmsSender} interface, store it in the local {\f1\fs20 SmsSender} variable, and let its {\f1\fs20 Send} method return {\f1\fs20 true}.
What is nice with this subbing / mocking implementation is that:
- The "fluent" style of coding makes it easy to write and read the class behavior, without any actual coding in {\i Delphi}, nor class definition;
- Even if {\f1\fs20 ISmsSender} has a lot of methods, only {\f1\fs20 Send} matters for us: {\f1\fs20 TInterfaceStub} will create all those methods, and let them return default values, with additional line of code needed;
- Memory allocation will be handled by the framework: when {\f1\fs20 SmsSender} instance will be released, the associated {\f1\fs20 TInterfaceStub} data will also be freed (and in case a mock, any expectations will be verified).
:   Defining a mock
Now we will define another fake class, which may fail the test, so it is called a "@*mock@", and the {\i mORMot} generator class will be {\f1\fs20 TInterfaceMock}:
!  TInterfaceMock.Create(TypeInfo(IUserRepository),UserRepository,self).
!    ExpectsCount('Save',qoEqualTo,1);
We provide the {\f1\fs20 TMyTest} instance as {\f1\fs20 self} to the {\f1\fs20 TInterfaceMock constructor}, to associate the mocking aspects with this test case. That is, any registered {\f1\fs20 Expects*()} rule will let {\f1\fs20 TMyTest.Check()} be called with a {\f1\fs20 boolean} condition reflecting the test validation status of every rule.
The {\f1\fs20 ExpectsCount()} method is indeed where mocking is defined. When the {\f1\fs20 UserRepository} generated instance is released, {\f1\fs20 TInterfaceMock} will check all the {\f1\fs20 Expects*()} rules, and, in this case, check that the {\f1\fs20 Save} method has been called exactly one time ({\f1\fs20 qoEqualTo,1}).
:   Running the test
Since we have all the expected stub and mock at hand, let's run the test itself:
!  with TLoginController.Create(UserRepository,SmsSender) do
!  try
!    ForgotMyPassword('toto');
!  finally
!    Free;
!  end;
That is, we run the actual implementation method, which will call our fake methods:
!procedure TLoginController.ForgotMyPassword(const UserName: RawUTF8);
!var U: TUser;
!begin
!  U := fUserRepository.GetUserByName(UserName);
!  U.Password := Int32ToUtf8(Random(MaxInt));
!  if fSmsSender.Send('Your new password is '+U.Password,U.MobilePhoneNumber) then
!    fUserRepository.Save(U);
!end;
Let's put all this together.
\page
: Stubs and Mocks in mORMot
Our {\i mORMot} framework is therefore able to @*stub@ or @*mock@ any {\i Delphi} {\f1\fs20 interface}.
We will now detail how it is expected to work.
:141  Direct use of interface types without TypeInfo()
First of all, it is a good practice to always register your service interfaces in the unit which define their type, as such:
!unit MyServiceInterfaces;
!...
!type
!  ISmsSender = interface(IInvokable)
!  ...
!  IUserRepository = interface(IInvokable)
!  ...
!
!initialization
!  TInterfaceFactory.RegisterInterfaces(
!    [TypeInfo(ISmsSender),TypeInfo(IUserRepository)]);
!end.
Then creating a stub or a mock could be done directly from the interface name, which will be transmitted as its {\f1\fs20 TGUID}, without the need of using the {\f1\fs20 TypeInfo()} pseudo-function:
!  TInterfaceStub.Create(ISmsSender,SmsSender);
!  TInterfaceMock.Create(IUserRepository,UserRepository,self);
In the code below, we will assume that the {\f1\fs20 interface} type information has been registered, so that we may be able to use directly {\f1\fs20 I*} without the {\f1\fs20 TypeInfo(I*)} syntax
:  Manual dependency injection
As usual, the best way to explain what a library does is to look at the code using it.
Here is an example (similar to the one shipped with {\i RhinoMocks}) of verifying that when we execute the "forgot my password" scenario as implemented by the {\f1\fs20 TLoginController} class, we actually called the {\f1\fs20 Save()} method:
!procedure TMyTest.ForgotMyPassword;
!var SmsSender: ISmsSender;
!    UserRepository: IUserRepository;
!begin
!  TInterfaceStub.Create(ISmsSender,SmsSender).
!    Returns('Send',[true]);
!  TInterfaceMock.Create(IUserRepository,UserRepository,self).
!    ExpectsCount('Save',qoEqualTo,1);
!  with TLoginController.Create(UserRepository,SmsSender) do
!  try
!    ForgotMyPassword('toto');
!  finally
!    Free;
!  end;
!end;
And... that's all, since the verification will take place when {\f1\fs20 IUserRepository} instance will be released.
If you want to follow the "test spy" pattern (i.e. no expectation defined {\i a priori}, but manual check after the execution), you can use:
!procedure TMyTest.ForgotMyPassword;
!var SmsSender: ISmsSender;
!    UserRepository: IUserRepository;
!    Spy: TInterfaceMockSpy;
!begin
!  TInterfaceStub.Create(ISmsSender,SmsSender).
!    Returns('Send',[true]);
!  Spy := TInterfaceMockSpy.Create(IUserRepository,UserRepository,self);
!  with TLoginController.Create(UserRepository,SmsSender) do
!  try
!    ForgotMyPassword('toto');
!  finally
!    Free;
!  end;
!  Spy.Verify('Save');
!end;
This is something unique with our library: you can decide if you want to use the classic "expect-run-verify" pattern, or the somewhat more direct "run-verify" / "test spy" pattern. With {\i mORMot}, you pick up your mocking class (either {\f1\fs20 TInterfaceMock} or {\f1\fs20 TInterfaceMockSpy}), then use it as intended. You can even mix the two aspects in the same instance! It is just a matter of taste and opportunity for you to use the right pattern.
For another easier pattern, like the one in the {\i Mockito} home page:
!  TInterfaceMock.Create(ICalculator,ICalc,self).
!    ExpectsCount('Multiply',qoEqualTo,1).
!    ExpectsCount('Add',[10,20],qoEqualTo,1);
!  ICalc.Add(10,20);
!  ICalc.Multiply(10,30)
If you want to follow the "test spy" pattern, you can use:
!  Mock := TInterfaceMockSpy.Create(ICalculator,ICalc,self);
!  ICalc.Add(10,20);
!  ICalc.Multiply(10,30)
!  Mock.Verify('Add');
!  Mock.Verify('Multiply',[10,30]);
If you compare with existing mocking frameworks, even in other languages / platforms like the two above, you will find out that the features included in {\i mORMot} are quite complete:
- Stubbing of any method, returning default values for results;
- Definition of the stubbed behavior via a simple fluent interface, with {\f1\fs20 TInterfaceStub.Returns()}, including easy definition of returned results values, for the whole method or following parameters/arguments matchers;
- Handle methods with {\f1\fs20 var, out} or function result returned values - i.e. not only the function result (as other {\i Delphi} implementations does, due to a limitation of the {\f1\fs20 TVirtualInterface} standard implementation, on which {\i mORMot} does not rely), but all outgoing values, as an array of values;
- Stubbed methods can use delegates or event callbacks with {\f1\fs20 TInterfaceStub.Executes()} rule definitions, for the whole method or following parameters/arguments matchers,  to run a more complex process;
- Stubbed methods can also raise exceptions with {\f1\fs20 TInterfaceStub.Raises()} rule definitions, for the whole method or following parameters/arguments matchers, if this is the behavior to be tested;
- Clear distinction between {\i mocks} and {\i stubs}, with two dedicated classes, named {\f1\fs20 TInterfaceStub} and {\f1\fs20 TInterfaceMock};
- Mocks are directly linked to {\i mORMot}'s unitary tests / test-driven classes - see @12@;
- Mocked methods can trigger test case failure with {\f1\fs20 TInterfaceMock.Fails()} definitions, for the whole method or following parameters/arguments matchers;
- Mocking via "expect-run-verify" or "run-verify" (aka "test spy") patterns, on choice, depending on your testing expectations;
- Mocking validation against number of execution of a method, or a method with arguments/parameters matchers, or the global execution trace - in this case, pass count can be compared with operators like {\f1\fs20 < <= = <> > >=} and not only the classic exact-number-of-times and at-least-once verifications;
- Most common parameters and results can be defined as simple {\f1\fs20 array of const} in the {\i Delphi} code, or by supplying JSON arrays (needed e.g. for more complex structures like {\f1\fs20 record} values);
- Execution trace retrieval in easy to read or write text format (and not via complex "fluent" interface e.g. with {\f1\fs20 When} clauses);
- Auto-release of the {\f1\fs20 TInterfaceStub TInterfaceMock TInterfaceMockSpy} generator instance, when the interface is no longer required, to minimize the code to type, and avoid potential memory leaks;
- Works from {\i Delphi} 6 up to the latest available {\i Delphi} version - since no use of syntax sugar like generics, nor the {\f1\fs20 RTTI.pas} features;
- Very good performance (the faster {\i Delphi} mocking framework, for sure), due to very low overhead and its reuse of {\i mORMot}'s low-level interface-based services kernel using JSON serialization, which does not rely on the slow and limited {\f1\fs20 TVirtualInterface}.
:  Stubbing complex return values
Just imagine that the {\f1\fs20 ForgotMyPassword} method does perform an internal test:
!procedure TLoginController.ForgotMyPassword(const UserName: RawUTF8);
!var U: TUser;
!begin
!  U := fUserRepository.GetUserByName(UserName);
!!  Assert(U.Name=UserName);
!  U.Password := Int32ToUtf8(Random(MaxInt));
!  if fSmsSender.Send('Your new password is '+U.Password,U.MobilePhoneNumber) then
!    fUserRepository.Save(U);
!end;
This will fail the test for sure, since by default, {\f1\fs20 GetUserByName} stubbed method will return a valid but void record. It means that {\f1\fs20 U.Name} will equal {\f1\fs20 ''}, so the highlighted line will raise an {\f1\fs20 EAssertionFailed} exception.
Here is how we may enhance our stub, to ensure it will return a {\f1\fs20 TUser} value matching {\f1\fs20 U.Name='toto'}:
!var UserRepository: IUserRepository;
!    U: TUser;
!  (...)
!!  U.Name := 'toto';
!  TInterfaceMock.Create(IUserRepository,UserRepository,self).
!!    Returns('GetUserByName','"toto"',RecordSaveJSON(U,TypeInfo(TUser))).
!    ExpectsCount('Save',qoEqualTo,1);
The only trick in the above code is that we use {\f1\fs20 RecordSaveJSON()} function to compute the internal JSON representation of the record, as expected by {\i mORMot}'s data marshalling.
:  Stubbing via a custom delegate or callback
In some cases, it could be very handy to define a complex process for a given method, without the need of writing a whole implementation class.
A delegate or event callback can be specified to implement this process, with three parameters marshalling modes:
- Via some {\f1\fs20 Named[]} variant properties (which are the default for the {\f1\fs20 Ctxt} callback parameter) - the easiest and safest to work with;
- Via some {\f1\fs20 Input[]} and {\f1\fs20 Output[]} variant properties;
- Directly as a JSON array text (the fastest, since native to the {\i mORMot} core).
Let's emulate the following behavior:
!function TServiceCalculator.Subtract(n1, n2: double): double;
!begin
!  result := n1-n2;
!end;
:   Delegate with named variant parameters
You can stub a method using a the {\f1\fs20 Named[] variant} arrays as such:
!  TInterfaceStub.Create(ICalculator,ICalc).
!!    Executes('Subtract',IntSubtractVariant);
!  (...)
!  Check(ICalc.Substract(10.5,1.5)=9);
The callback {\f1\fs20 function} can be defined as such:
!procedure TTestServiceOrientedArchitecture.IntSubtractVariant(
!  Ctxt: TOnInterfaceStubExecuteParamsVariant);
!begin
!!  Ctxt['result'] := Ctxt['n1']-Ctxt['n2'];
!end;
That is, callback shall use {\f1\fs20 Ctxt['']} property to access the parameters and result as {\f1\fs20 variant} values.
In fact, we use the {\f1\fs20 Ctxt.Named[] default} property, so it is exactly as the following line:
!  Ctxt.Named['result'] := Ctxt.Named['n1']-Ctxt.Named['n2'];
If the execution fails, it shall execute {\f1\fs20 Ctxt.Error()} method with an associated error message to notify the stubbing process of such a failure.
Using named parameters has the advantage of being more explicit in case of change of the method signature (e.g. if you add or rename a parameter). It should be the preferred way of implementing such a callback, in most cases.
:   Delegate with indexed variant parameters
There is another way of implementing such a callback method, directly by using the {\f1\fs20 Input[]} and {\f1\fs20 Output[]} indexed properties. It should be (a bit) faster to execute:
!procedure TTestServiceOrientedArchitecture.IntSubtractVariant(
!  Ctxt: TOnInterfaceStubExecuteParamsVariant);
!begin
!  with Ctxt do
!    Output[0] := Input[0]-Input[1]; // result := n1-n2
!end;
Just as with {\f1\fs20 TOnInterfaceStubExecuteParamsJSON} implementation, {\f1\fs20 Input[]} index follows the exact order of {\f1\fs20 const} and {\f1\fs20 var} parameters at method call, and {\f1\fs20 Output[]} index follows the exact order of {\f1\fs20 var} and {\f1\fs20 out} parameters plus any {\f1\fs20 function} result.
That is, if you call:
!  function Subtract(n1,n2: double): double;
! ...
!  MyStub.Substract(100,20);
you have in {\f1\fs20 TOnInterfaceStubExecuteParamsJSON}:
! Ctxt.Params = '100,20.5'; // at method call
! Ctxt.Result = '[79.5]';   // after Ctxt.Returns([..])
and in the {\f1\fs20 variant} arrays:
! Ctxt.Input[0] = 100;      // =n1 at method call
! Ctxt.Input[1] = 20.5;     // =n2 at method call
! Ctxt.Output[0] = 79.5;    // =result after method call
In case of additional {\f1\fs20 var} or {\f1\fs20 out} parameters, those should be added to the {\f1\fs20 Output[]} array before the last one, which is always the function result.
If the method is defined as a {\f1\fs20 procedure} and not as a {\f1\fs20 function}, of course there is no last {\f1\fs20 Output[]} item, but only {\f1\fs20 var} or {\f1\fs20 out} parameters.
:   Delegate with JSON parameters
You can stub a method using a JSON array as such:
!  TInterfaceStub.Create(ICalculator,ICalc).
!!    Executes('Subtract',IntSubtractJSON);
!  (...)
!  Check(ICalc.Substract(10.5,1.5)=9);
The callback shall be defined as such:
!procedure TTestServiceOrientedArchitecture.IntSubtractJSON(
!  Ctxt: TOnInterfaceStubExecuteParamsJSON);
!var P: PUTF8Char;
!begin // result := n1-n2
!!  P := pointer(Ctxt.Params);
!!  Ctxt.Returns([GetNextItemDouble(P)-GetNextItemDouble(P)]);
!end;
That is, it shall parse incoming parameters from {\f1\fs20 Ctxt.Params}, and store the result values as a JSON array in {\f1\fs20 Ctxt.Result}.
Input parameter order in {\f1\fs20 Ctxt.Params} follows the exact order of {\f1\fs20 const} and {\f1\fs20 var} parameters at method call, and output parameter order in {\f1\fs20 Ctxt.Returns([])} or {\f1\fs20 Ctxt.Result} follows the exact order of {\f1\fs20 var} and {\f1\fs20 out} parameters plus any {\f1\fs20 function} result.
This method could have been written as such, if you prefer to return directly the JSON array:
!procedure TTestServiceOrientedArchitecture.IntSubtractJSON(
!  Ctxt: TOnInterfaceStubExecuteParamsJSON);
!var P: PUTF8Char;
!begin // result := n1-n2
!!  P := pointer(Ctxt.Params);
!!  Ctxt.Result := '['+DoubleToStr(GetNextItemDouble(P)-GetNextItemDouble(P))+']';
!end;
This may sound somewhat convenient here in case of double values, but it will be error prone if types are more complex. In all cases, using {\f1\fs20 Ctxt.Returns([])} is the preferred method.
:   Accessing the test case when mocking
In case of mocking, you may add additional verifications within the implementation callback, as such:
!  TInterfaceMock.Create(ICalculator,ICalc,self).
!    Executes('Subtract',IntSubtractVariant,'toto');
! (...)
!procedure TTestServiceOrientedArchitecture.IntSubtractVariant(
!  Ctxt: TOnInterfaceStubExecuteParamsVariant);
!begin
!!  Ctxt.TestCase.Check(Ctxt.EventParams='toto');
!  Ctxt['result'] := Ctxt['n1']-Ctxt['n2'];
!end;
Here, an additional callback-private parameter containing {\f1\fs20 'toto'} has been specified at {\f1\fs20 TInterfaceMock} definition. Then its content is checked on the associated test case via {\f1\fs20 Ctxt.Sender} instance. If the caller is not a {\f1\fs20 TInterfaceMock}, it will raise an exception when accessing the {\f1\fs20 Ctxt.TestCase} property.
:180  Calls tracing
As stated above, {\i mORMot} is able to log all interface calls into its internal {\f1\fs20 TInterfaceStub}'s structures. This is indeed the root feature of its "test spy" {\f1\fs20 TInterfaceMockSpy.Verify()} methods.
!  Stub := TInterfaceStub.Create(ICalculator,I).
!!    SetOptions([imoLogMethodCallsAndResults]);
!  Check(I.Add(10,20)=0,'Default result');
!!  Check(Stub.LogAsText='Add(10,20)=[0]');
Here above, we retrieved the whole call stack, including input parameters and returned results, as an easy-to-read JSON content. We found out that JSON is a very convenient way of tracing the method calls, both efficient for the computer and the human being hardly testing the code.
A more complex trace verification could be defined for instance, in the context of an interface {\i mock}:
!  TInterfaceMock.Create(ICalculator,I,self).
!    Returns('Add','30').
!    Returns('Multiply',[60]).
!    Returns('Multiply',[2,35],[70]).
!    ExpectsCount('Multiply',qoEqualTo,2).
!    ExpectsCount('Subtract',qoGreaterThan,0).
!    ExpectsCount('ToTextFunc',qoLessThan,2).
!    // check trace for a whole method execution
!!    ExpectsTrace('Add','Add(10,30)=[30]').
!!    ExpectsTrace('Multiply','Multiply(10,30)=[60],Multiply(2,35)=[70]').
!    // check trace for a whole method execution, filtering with given parameters
!!    ExpectsTrace('Multiply',[10,30],'Multiply(10,30)=[60]').
!    // check trace for the whole interface execution
!!    ExpectsTrace('Add(10,30)=[30],Multiply(10,30)=[60],'+
!!      'Multiply(2,35)=[70],Subtract(2.3,1.2)=[0],ToTextFunc(2.3)=["default"]').
!    Returns('ToTextFunc',['default']);
!  Check(I.Add(10,30)=30);
!  Check(I.Multiply(10,30)=60);
!  Check(I.Multiply(2,35)=70);
!  Check(I.Subtract(2.3,1.2)=0,'Default result');
!  Check(I.ToTextFunc(2.3)='default');
The overloaded {\f1\fs20 ExpectsTrace()} methods are able to add some checks not only about the number of calls of a given method, but the exact order of the executed commands, with associated parameters and all retrieved result values. They can validate the trace of one specific method (optionally with a filter against the incoming parameters), or globally for the whole mocked interface.
Note that internally, those methods will compute a {\f1\fs20 Hash32()} hash value of the expected trace, which is a good way of minimizing data in memory or re-use a value retrieved at execution time for further regression testing. Some {\i overloaded} signatures are indeed available to directly specify the expected {\f1\fs20 Hash32()} value, in case of huge regression scenarios: run the test once, debugging all expected behavior by hand, then store the hash value to ensure that no expected step will be broken in the future.
You have even a full access to the internal execution trace, via the two {\f1\fs20 TInterfaceStub.Log} and {\f1\fs20 LogCount} properties. This will allow any validation of mocked {\f1\fs20 interface} calls logic, beyond {\f1\fs20 ExpectsTrace()} possibilities.
You can take a look at {\f1\fs20 TTestServiceOrientedArchitecture.MocksAndStubs} regression tests, for a whole coverage of all the internal features.
\page
:161 Dependency Injection and Interface Resolution
In our example, we {\i injected} the dependencies explicitly as parameters to the class {\f1\fs20 constructor} - see @162@. We will present @63@, in a dedicated chapter, how the framework @*SOA@ features do resolve services as interfaces.
But real-world application may be much complex, and a generic way of resolving dependencies, and {\i @*Inversion Of Control@} (aka @*IoC@) has been implemented.
First of all, if you inherit from {\f1\fs20 TInjectableObject}, you will be able to resolve dependencies in two ways:
- Explicitly via its {\f1\fs20 Resolve()} overloaded methods, for lazy initialization of any registered {\f1\fs20 interface};
- Automatically at instance creation, for all its {\f1\fs20 published} properties declared with an {\f1\fs20 interface} type.
A dedicated set of overloaded constructors is also available at {\f1\fs20 TInjectableObject class} level, so that you may be able to easily stub/mock or inject any instance, e.g. for testing purposes:
!procedure TMyTestCase.OneTestCaseMethod;
! var Test: IServiceToBeTested;
! begin
!!   Test := TServiceToBeTested.CreateInjected(
!     [ICalculator],
!     [TInterfaceMock.Create(IPersistence,self).
!       ExpectsCount('SaveItem',qoEqualTo,1),
!      RestInstance.Services],
!     [AnyInterfacedObject]);
!   ...
In this code, we have @141@. So we could write directly {\f1\fs20 ICalculator} or {\f1\fs20 IPersistence} to refer to an explicit {\f1\fs20 interface} type.
This test case ({\f1\fs20 TMyTestCase} inherits from {\f1\fs20 TSynTestCase}) will create a {\f1\fs20 TServiceToBeTested} instance, create a {\f1\fs20 TInterfaceStub} for its {\f1\fs20 ICalculator} dependency, then a {\f1\fs20 TInterfaceMock} expecting the {\f1\fs20 IPersistence.SaveItem} method to be called exactly one time, allowing resolution from a {\f1\fs20 TSQLRest.Services} SOA resolver, and injecting a pre-existing {\f1\fs20 AnyInterfacedObject TInterfacedObject} instance.
Then, dependency resolution may take place as {\f1\fs20 published} properties:
!type
!  TServiceToBeTested = class(TInjectableObject)
!  protected
!    fCalculator: ICalculator;
!  ...
!!  published
!!    property Calculator: ICalculator read fCalculator;
!    ...
!  end;
!
! ...
!
!function TServiceToBeTested.DoCalculation(a,b: integer): integer;
!begin
!  result := Calculator.Add(a,b);
!end;
This {\f1\fs20 fCalculator} instance will be resolved and instantiated by {\f1\fs20 TInjectableObject.Create}, then released as any regular {\f1\fs20 interface} field in the {\f1\fs20 class} {\f1\fs20 destructor}. You do not have to overload the {\f1\fs20 TServiceToBeTested constructor}, nor manage this {\f1\fs20 fCalculator} life time. Its auto-created instance will be shared by the whole {\f1\fs20 TServiceToBeTested} context, so it should be either stateless (like adding two numbers), or expected to evolve at each use.
Sometimes, there may be an over-cost to initialize such properties each time a {\f1\fs20 TServiceToBeTested class} instance is created. Or maybe the {\f1\fs20 interface} implementation is not stateless, and a new instance should be retrieved before each use. As an alternative, any {\f1\fs20 interface} may be resolved on need, in a {\i lazy} way:
!procedure TServiceToBeTested.DoSomething;
!var persist: IPersistence;
!begin
!!  Resolve(IPersistence,persist);
!  persist.SaveItem('John','Doe');
!end;
The {\f1\fs20 TInjectableObject.Resolve()} overloaded methods will retrieve one instance of the asked {\f1\fs20 interface}. The above code will raise an exception if the supplied {\f1\fs20 IPersistence} was not previously registered to the {\f1\fs20 TInjectableObject class}.
When such an {\f1\fs20 TInjectableObject} instance is created within {\i mORMot}'s SOA methods (i.e. {\f1\fs20 TSQLRest.Services} property), the injection will transparently involve all registered classes. Also take a look at the {\f1\fs20 TInterfaceResolverInjected.RegisterGlobal()} overloaded methods, which are able to register some {\f1\fs20 class} types or instances globally for the whole executable context. Just make sure that you won't break the @158@, by defining such a global registration, which should occur only for specific needs, truly orthogonal to the whole application, or specific to a test case.
\page
:63Client-Server services via interfaces
%cartoon06.png
In real world, especially when your application relies heavily on services, the @49@ implementation pattern has some drawbacks:
- Most content marshalling is to be done by hand, so may introduce implementation issues;
- Client and server side code does not have the same implementation pattern, so you will have to code explicitly data marshalling twice, for both client and server ({\i DataSnap} and WCF both suffer from a similar issue, by which client classes shall be coded separately, most time generated by a Wizard);
- You can not easily {\i test} your services, unless you write a lot of code to emulate a "fake" service implementation;
- The services do not have any hierarchy, and are listed as a plain list, which is not very convenient;
- It is difficult to synchronize several service calls within a single context, e.g. when a workflow is to be handled during the application process (you have to code some kind of state machine on both sides, and define all session handling by hand);
- @*Security@ is handled globally for the user, or should be checked by hand in the implementation method (using the {\f1\fs20 Ctxt.Session*} members);
- There is no way of implementing service {\i callbacks}, using e.g. {\i @*WebSockets@}.
You can get rid of those limitations with the interface-based service implementation of {\i mORMot}. For a detailed introduction and best practice guide to @*SOA@, see @17@. All commonly expected SOA features are now available in the current implementation of the {\i mORMot} framework (including service catalog aka "broker", via the optional publication of {\f1\fs20 interface} signatures).
\page
: Implemented features
Here are the key features of the current implementation of services using interfaces in the {\i Synopse mORMot framework}, as implemented in @!Lib\mORMot.pas@ unit:
|%25%75
|\b Feature|Remarks\b0
|Service Orientation|Allow loosely-coupled relationship
|Design by contract|Service Contracts are defined in {\i Delphi} code as standard {\f1\fs20 interface} custom types
|Factory driven|Get an implementation instance from a given interface
|Server factory|You can get an implementation on the server side
|Client factory|You can get a "fake" implementation on the client side, remotely calling the server to execute the process
|Cross-platform clients|A {\i mORMot} server is able to generate cross-platform client code via a set of templates - see @86@
|Auto marshalling|The contract is transparently implemented: no additional code is needed e.g. on the client side, and will handle simple types (strings, numbers, dates, sets and enumerations) and high-level types (objects, collections, records, dynamic arrays, variants) from {\i Delphi} 6 up to the latest available {\i Delphi} version
|Flexible|Methods accept per-value or per-reference parameters
|Instance lifetime|An implementation class can be:\line - Created on every call,\line - Shared among all calls,\line - Shared for a particular user or group,\line - Dedicated to the thread it runs on,\line - Alive as long as the client-side interface is not released,\line - Or as long as an @*authentication@ session exists
|@*Stateless@|Following a standard request/reply pattern
|Statefull|Server side implementation may be synchronized with client-side interface, e.g. over {\i @*WebSockets@}
|Dual way|You can define callbacks, using e.g. {\i @*WebSockets@} for immediate notification
|Signed|The contract is checked to be consistent before any remote execution
|Secure|Every service and/or methods can be enabled or disabled on need
|Safe|Using extended RESTful authentication - see @18@
|Multi-hosted\line (with DMZ)|Services are hosted by default within the main @*ORM@ server, but can have their own process, with a dedicated connection to the ORM core
|Broker ready|Service meta-data can be optionally revealed by the server
|Multiple transports|All Client-Server protocols of {\i mORMot} are available, i.e. direct in-process connection, Windows Messages, named pipes, TCP/IP-HTTP
|JSON based|Transmitted data uses {\i JavaScript Object Notation}
|Routing choice|Services are identified either at the URI level (the @*REST@ful way), or in a @*JSON-RPC@ model (the @*AJAX@ way), or via any custom format (using {\f1\fs20 class} inheritance)
|AJAX and RESTful|JSON and HTTP combination allows services to be consumed from AJAX rich clients
|Light & fast|Performance and memory consumption are very optimized, in order to ensure scalability and ROI
|%
: How to make services
The typical basic tasks to perform are the following:
- Define the service contract;
- Implement the contract;
- Configure and host the service;
- Build a client application.
We will describe those items.
\page
: Defining a service contract
In a @*SOA@, services tend to create a huge list of operations. In order to facilitate implementation and maintenance, operations shall be grouped within common services.
Before defining how such services are defined within {\i mORMot}, it is worth applying the @17@ main principles, i.e. loosely-coupled relationship. When you define {\i mORMOt} contracts, ensure that this contract will stay un-coupled with other contracts. It will help writing @*SOLID@ code, enhance maintenability, and allow introducing other service providers on demand (some day or later, you'll certainly be asked to replace one of your service with a third-party existing implementation of the corresponding feature: you shall at least ensure that your own implementation will be easily re-coded with external code, using e.g. a @*SOAP@/WSDL @*gateway@).
:  Define an interface
The service contract is to be defined as a plain {\i Delphi} {\f1\fs20 interface} type. In fact, the sample type as stated above - see @46@ - can be used directly:
!type
!  ICalculator = interface(IInvokable)
!    ['{9A60C8ED-CEB2-4E09-87D4-4A16F496E5FE}']
!    /// add two signed 32-bit integers
!    function Add(n1,n2: integer): integer;
!  end;
This {\f1\fs20 ICalculator.Add} method will define one "{\i Add}" operation, under the "{\i ICalculator}" service (which will be named internally {\f1\fs20 'Calculator'} by convention). This operation will expect two numbers as input, and then return the sum of those numbers.
The current implementation of service has the following expectations:
- Any interface inheriting from {\f1\fs20 IInvokable}, with a @*GUID@, can be used - we expect the RTTI to be available, so {\f1\fs20 IInvokable} is a good parent type;
- You can inherit an interface from an existing one: in this case, the inherited methods will be part of the child interface, and will be expected to be implemented (just as with standard {\i Delphi} code);
- Only plain ASCII names are allowed for the type definition (as it is conventional to use English spelling for service and operation naming);
- Calling convention shall be {\f1\fs20 register} (the {\i Delphi}'s default) - nor {\f1\fs20 stdcall} nor {\f1\fs20 cdecl} is available yet, but this won't be a restriction since the {\f1\fs20 interface} definition is dedicated to {\i Delphi} code scope;
- Methods can have a result, and accept per-value or per-reference parameters.
In fact, parameters expectations are the following:
- Simple types (strings, numbers, dates, sets and enumerations) and high-level types (objects, collections, records and dynamic arrays) are handled - see below for the details;
- They can be defined as {\f1\fs20 const}, {\f1\fs20 var} or {\f1\fs20 out} - in fact, {\f1\fs20 const} and {\f1\fs20 var} parameters values will be sent from the client to the server as JSON, and {\f1\fs20 var} and {\f1\fs20 out} parameters values will be returned as JSON from the server;
- {\f1\fs20 procedure} or {\f1\fs20 function} kind of method definition are allowed;
- Only exception is that you can't have a function returning a {\f1\fs20 class} instance (how will know when to release the instance in this case?), but such instances can be passed as {\f1\fs20 const}, {\f1\fs20 var} or {\f1\fs20 out} parameters (and {\f1\fs20 published} properties will be serialized within the JSON message);
- In fact, the {\f1\fs20 @*TCollection@} kind of parameter is not directly handled by the framework: you shall define a {\f1\fs20 @*TInterfacedCollection@} class, overriding its {\f1\fs20 GetClass} abstract virtual method (otherwise the server side won't be able to create the kind of collection as expected);
- Special {\f1\fs20 @*TServiceCustomAnswer@} kind of record can be used as {\f1\fs20 function} result to specify a custom content (with specified encoding, to be used e.g. for @*AJAX@ or HTML consumers) - in this case, no {\f1\fs20 var} nor {\f1\fs20 out} parameters values shall be defined in the method (only the BLOB value is returned).
:154  Service Methods Parameters
Handled types of parameters are:
|%30%70
|\b Delphi type|Remarks\b0
|{\f1\fs20 boolean}|Transmitted as @*JSON@ true/false
|{\f1\fs20 integer cardinal Int64 double currency}|Transmitted as JSON numbers
|enumerations|Transmitted as JSON number
|set|Transmitted as JSON number - one bit per element (up to 32 elements)
|{\f1\fs20 @*TDateTime@ @*TDateTimeMS@}|Transmitted as @*ISO 8601@ JSON text
|{\f1\fs20 @*RawUTF8@ @*WideString@ @*SynUnicode@}|Transmitted as JSON text (@*UTF-8@ encoded)
|{\f1\fs20 string}|Transmitted as UTF-8 JSON text, but prior to {\i Delphi} 2009, the framework will ensure that both client and server sides use the same ANSI code page - so you should better use {\f1\fs20 RawUTF8} everywhere
|{\f1\fs20 @*RawJSON@}|UTF-8 buffer transmitted with no serialization (wheras a {\f1\fs20 RawUTF8} will be escaped as a JSON string) - expects to contain valid JSON content, e.g. for TSQLTableJSON requests
|{\f1\fs20 @*RawByteString@}|Transmitted as @*Base64@ encoded JSON text - see @197@ for optional binary transmission
|{\f1\fs20 @*TPersistent@}|Published properties will be transmitted as JSON object
|{\f1\fs20 @*TSQLRecord@}|All published fields (including ID) will be transmitted as JSON object
|{\f1\fs20 @*TCollection@}|Not allowed direcly: inherit from {\f1\fs20 @*TInterfacedCollection@} or call {\f1\fs20 TJSONSerializer. RegisterCollectionForJSON()}
|{\f1\fs20 @*TInterfacedCollection@}|Transmitted as a JSON array of JSON objects - see @55@
|{\f1\fs20 @*TObjectList@}|Transmitted as a JSON array of JSON objects, with a {\f1\fs20 "ClassName": "TMyClass"} field to identify the type - see @71@
|any {\f1\fs20 @*TObject@}|See @52@
|dynamic arrays|Transmitted as JSON arrays - see @48@
|{\f1\fs20 @*record@}|Need to have RTTI (so a string or dynamic array field within), just like with regular {\i Delphi} {\f1\fs20 interface} expectations - transmitted as binary with @*Base64@ encoding before {\i Delphi} 2010, or as JSON object thanks to the @*enhanced RTTI@ available since, or via an custom JSON serialization - see @51@
|{\f1\fs20 variant}|Transmitted as JSON, with support of @80@ for objects and arrays; OLE {\f1\fs20 variant} arrays are not handled: use {\f1\fs20 _Arr([]) _ArrFast([])} instead
|{\f1\fs20 @*TServiceCustomAnswer@}|If used as a {\f1\fs20 function} result (not as parameter), the supplied content will be transmitted directly to the client (with no JSON @*serialization@); in this case, no {\f1\fs20 var} nor {\f1\fs20 out} parameters are allowed in the method - it will be compatible with both our {\f1\fs20 TServiceFactoryClient} implementation, and any other service consumers (e.g. @*AJAX@)
|{\f1\fs20 interface}|A callback instance could be specified, to allow asynchronous notification, using e.g. {\i WebSockets} - see @149@
|%
You can therefore define complex {\f1\fs20 interface} types, as such:
!type
!  ICalculator = interface(IInvokable)
!    ['{9A60C8ED-CEB2-4E09-87D4-4A16F496E5FE}']
!    /// add two signed 32-bit integers
!    function Add(n1,n2: integer): integer;
!    /// multiply two signed 64-bit integers
!    function Multiply(n1,n2: Int64): Int64;
!    /// substract two floating-point values
!    function Subtract(n1,n2: double): double;
!    /// convert a currency value into text
!    procedure ToText(Value: Currency; var Result: RawUTF8);
!    /// convert a floating-point value into text
!    function ToTextFunc(Value: double): string;
!    /// do some work with strings, sets and enumerates parameters,
!    // testing also var (in/out) parameters and set as a function result
!    function SpecialCall(Txt: RawUTF8; var Int: integer; var Card: cardinal; field: TSynTableFieldTypes;
!      fields: TSynTableFieldTypes; var options: TSynTableFieldOptions): TSynTableFieldTypes;
!    /// test integer, strings and wide strings dynamic arrays, together with records
!    function ComplexCall(const Ints: TIntegerDynArray; Strs1: TRawUTF8DynArray;
!      var Str2: TWideStringDynArray; const Rec1: TVirtualTableModuleProperties;
!      var Rec2: TSQLRestCacheEntryValue): TSQLRestCacheEntryValue;
!    /// test variant kind of parameters
!    function TestVariants(const Text: RawUTF8; V1: variant; var V2: variant): variant;
!    /// validates ArgsInputIsOctetStream raw binary upload
!    function DirectCall(const Data: TSQLRawBlob): integer;
!  end;
Note how {\f1\fs20 SpecialCall} and {\f1\fs20 ComplexCall} methods have quite complex parameters definitions, including dynamic arrays, sets and records. {\f1\fs20 DirectCall} will use binary POST, by-passing @*Base64@ JSON encoding - see @197@. The framework will handle {\f1\fs20 const} and {\f1\fs20 var} parameters as expected, i.e. as input/output parameters, also on the client side. Any simple types of dynamic arrays (like {\f1\fs20 TIntegerDynArray}, {\f1\fs20 TRawUTF8DynArray}, or {\f1\fs20 TWideStringDynArray}) will be serialized as plain JSON arrays - the framework is able to handle any dynamic array definition, but will serialize those simple types in a more AJAX compatible way, thanks to the enhanced RTTI available since to {\i Delphi} 2010.
:  TPersistent / TSQLRecord parameters
As stated above, {\i mORMot} does not allow a method {\f1\fs20 function} to return a {\f1\fs20 class} instance.
That is, you can't define such a method:
!  ICustomerFactory = interface(IInvokable)
!    ['{770D009F-15F4-4307-B2AD-BBAE42FE70C0}']
!    function NewCustomer: TCustomer;
!  end;
Who will be in charge of freeing the instance, in client-server mode? There is no standard allocation scheme, in {\i Delphi}, for such parameters. So every {\f1\fs20 TObject} parameter instance shall be managed by the caller, i.e. allocated before the call and released after it. The method will just read or write the instance published properties, and serialize them as JSON.
What you can define is such a method:
!  ICustomerFactory = interface(IInvokable)
!    ['{770D009F-15F4-4307-B2AD-BBAE42FE70C0}']
!    procedure NewCustomer(out aCustomer: TCustomer);
!  end;
Note that here the {\f1\fs20 out} keyword does not indicate how the memory is allocated, but shows the communication direction of the remote service, i.e. it will serialize the object at method return. The caller shall instantiate an instance before call - whereas for "normal" {\i Delphi} code, it may be up to the method to instantiate the instance, and return it.
Then your client code can use it as such:
!var Factory: ICustomerFactory;
!    Repository: ICustomerRepository;
!    Customer: TCustomer;
!...
!  Customer := TCustomer.Create; // client side manage object instance
!  try
!    Customer.FirstName := StringToUTF8(EditFirstName.Text);
!    Customer.LastName := StringToUTF8(EditLastName.Text);
!    NewCutomerID := Repository.Save(Customer); // persist the object
!  finally
!    Customer.Free; // properly manage memory
!  end;
Or, using both {\i @*Factory@} and {\i @*Repository@} patterns, as proposed by @68@:
!var Factory: ICustomerFactory;
!    Repository: ICustomerRepository;
!    Customer: TCustomer;
!...
!  Factory.NewCustomer(Customer); // get a new object instance
!  try
!    Customer.FirstName := StringToUTF8(EditFirstName.Text);
!    Customer.LastName := StringToUTF8(EditLastName.Text);
!    NewCutomerID := Repository.Save(Customer); // persist the object
!  finally
!    Customer.Free; // properly manage memory
!  end;
In real live, it may be very easy to wrongly write a server method returning an existing instance, which will be released by the server SOA caller, and will trigger unexpected A/V randomly - very difficult to track - on the server side. Which is what we want to avoid... Whereas a pointer to nil gives always a clear access violation on the client side, which doesn't affect the server.\line So this requirement/limitation was designed as such to make the server side more resilient to errors, even if the client side is a bit more complex to work with. Usually, on the client side, you can safely pre-allocate your object instances, and reuse them.
:  Record parameters
By default, any {\f1\fs20 @*record@} parameter or function result will be serialized with a proprietary binary (and optimized) layout, then transmitted as a JSON string, after @*Base64@ encoding.
Even if older versions of {\i Delphi} are not able to generate the needed RTTI information for such serialization, allowing us only to use an efficient but proprietary binary layout, the {\i mORMot} framework offers a common way of implementing any custom serialization of records. See @51@.
Note that the callback signature used for {\i records} matches the one used for {\i dynamic arrays} serializations - see @53@ - as it will be shared between the two of them.
When records are used as {\i Data Transfer Objects} within services (which is a good idea in common SOA implementation patterns), such a custom serialization format can be handy, and makes more natural service consumption with AJAX clients.
:55  TCollection parameters
:   Use of TCollection
With {\i mORMot} services, you are able to define such a contract, e.g. for a {\f1\fs20 TCollTests} collection of {\f1\fs20 TCollTest} items:
!procedure Collections(Item: TCollTest; var List: TCollTests; out Copy: TCollTests);
Typical implementation of this contract may be:
!procedure TServiceComplexCalculator.Collections(Item: TCollTest;
!  var List: TCollTests; out Copy: TCollTests);
!begin
!  CopyObject(Item,List.Add);
!  CopyObject(List,Copy);
!end;
That is, it will append the supplied {\f1\fs20 Item} object to the provided {\f1\fs20 List} content, then return a copy in the {\f1\fs20 Copy} content:
- Setting {\f1\fs20 Item} without {\f1\fs20 var} or {\f1\fs20 out} specification is doing the same as {\f1\fs20 const}: it will be serialized from client to server (and not back from server to client);
- Setting {\f1\fs20 List} as {\f1\fs20 var} parameter will let this collection to be serialized from client to server, and back from server to the client;
- Setting {\f1\fs20 Copy} as {\f1\fs20 out} parameter will let this collection to be serialized only from server to client.
Note that {\f1\fs20 const / var / out} kind of parameters are used at the contract level in order to specify the direction of serialization, and not as usual (i.e. to define if it is passed {\i by value} or {\i by reference}). All {\f1\fs20 class} parameters shall be instantiated before method call: you can not pass any object parameter as nil (nor use it in a function result): it will raise an error.
Due to the current implementation pattern of the {\f1\fs20 TCollection} type in {\i Delphi}, it was not possible to implement directly this kind of parameter.
In fact, the {\f1\fs20 TCollection} constructor is defined as such:
! constructor Create(ItemClass: TCollectionItemClass);
And, on the server side, we do not know which kind of {\f1\fs20 TCollectionItemClass} is to be passed. Therefore, the {\f1\fs20 TServiceFactoryServer} is unable to properly instantiate the object instances, supplying the expected item class.
The framework propose two potential solutions:
- You can let your collection class inherit from the new {\f1\fs20 TInterfacedCollection} type;
- You can call the {\f1\fs20 TJSONSerializer.RegisterCollectionForJSON()} method to register the collection type and its associated item class.
We will now describe both ways.
:   Inherit from TInterfacedCollection
A dedicated {\f1\fs20 TInterfacedCollection} abstract type has been defined:
!  TInterfacedCollection = class(TCollection)
!  protected
!    class function GetClass: TCollectionItemClass; virtual; abstract;
!  public
!    constructor Create; reintroduce; virtual;
!  end;
In order to use a collection of objects, you will have to define at least the abstract method, for instance:
!  TCollTests = class(TInterfacedCollection)
!  protected
!    class function GetClass: TCollectionItemClass; override;
!  end;
!
!class function TCollTests.GetClass: TCollectionItemClass;
!begin
!  result := TCollTest;
!end;
Or, if you want a more complete / convenient implementation:
!  TCollTests = class(TInterfacedCollection)
!  private
!    function GetCollItem(Index: Integer): TCollTest;
!  protected
!    class function GetClass: TCollectionItemClass; override;
!  public
!    function Add: TCollTest;
!    property Item[Index: Integer]: TCollTest read GetCollItem; default;
!  end;
All other methods and properties (like {\f1\fs20 GetColItem / Add / Items[]}) are to be defined as usual.
:   Register a TCollection type
The other way of using {\f1\fs20 TCollection} kind of parameters is to declare it explicitly to the framework. You should call {\f1\fs20 JSONSerializer.RegisterCollectionForJSON()} with the corresponding {\f1\fs20 TCollection / TCollectionItem} class type pair.
Consider a dedicated class:
! TMyCollection = type(TCollection)
Note that a dedicated type is needed here. You just can't use this registration over a plain {\f1\fs20 TCollection}.
Then, for instance, after calling:
! TJSONSerializer.RegisterCollectionForJSON(TMyCollection,TMyCollectionItem);
The following lines of code are the same:
! MyColl := TMyCollection.Create(TMyCollectionItem);
! MyColl := ClassInstanceCreate(TMyCollection) as TMyCollection;
! MyColl := ClassInstanceCreate('TMyCollection') as TMyCollection;
The last two will retrieve the associated {\f1\fs20 TMyCollectionItem} class type from the previous registration.
Thanks to this internal registration table, {\i mORMot} will be able to serialize and unserialize plain {\f1\fs20 TCollection} type.
\page
: Server side
:  Implementing the service contract
In order to have an operating service, you'll need to implement a {\i Delphi} class which matches the expected {\f1\fs20 interface}.
In fact, the sample type as stated above - see @46@ - can be used directly:
!type
!  TServiceCalculator = class(TInterfacedObject, ICalculator)
!  public
!    function Add(n1,n2: integer): integer;
!  end;
!
!function TServiceCalculator.Add(n1, n2: integer): integer;
!begin
!  result := n1+n2;
!end;
And... That is all we need. The {\i Delphi} IDE will check at compile time that the class really implements the specified {\f1\fs20 interface} definition, so you'll be sure that your code meets the service contract expectations. Exact match (like handling type of parameters) will be checked by the framework when the service factory will be initialized, so you won't face any runtime exception due to a wrong definition.
Here the class inherits from {\f1\fs20 TInterfacedObject}, but you could use any plain {\i Delphi} class: the only condition is that it implements the {\f1\fs20 ICalculator} interface.
:  Set up the Server factory
In order to have a working service, you'll need to initialize a server-side @*factory@, as such:
! Server.ServiceRegister(TServiceCalculator,[TypeInfo(ICalculator)],sicShared);
You may prefer @141@, if the type has previously been registered:
! Server.ServiceDefine(TServiceCalculator,[ICalculator],sicShared);
The {\f1\fs20 Server} instance can be any {\f1\fs20 TSQLRestServer} inherited class, implementing any of the supported protocol of {\i mORMot}'s @35@, embedding a full {\i @*SQLite3@} engine (i.e. a {\f1\fs20 @*TSQLRestServerDB@} class) or a lighter in-memory engine (i.e. a {\f1\fs20 @*TSQLRestServerFullMemory@} class - which is enough for @*hosting@ services with authentication).
The code line above will register the {\f1\fs20 TServiceCalculator} class to implement the {\f1\fs20 ICalculator} service, with a single shared instance life time (specified via the {\f1\fs20 sicShared} parameter). An optional time out value can be specified, in order to automatically release a deprecated instance after some inactivity.
Whenever a service is executed, an implementation class is to be available. The life time of this implementation class is defined on both client and server side, by specifying a {\f1\fs20 TServiceInstanceImplementation} value. This setting must be the same on both client and server sides (it will be checked by the framework).
:92  Instances life time implementation
The available instance management options are the following:
|%24%76
|\b Lifetime|Description\b0
|{\f1\fs20 sicSingle}|One class instance is created per call:\line - This is the most expensive way of implementing the service, but is safe for simple workflows (like a one-type call);\line - This is the default setting for {\f1\fs20 TSQLRestServer.ServiceRegister}/{\f1\fs20 ServiceDefine} methods.
|{\f1\fs20 sicShared}|One object instance is used for all incoming calls and is not recycled subsequent to the calls
|{\f1\fs20 sicClientDriven}|One object instance will be created in synchronization with the client-side lifetime of the corresponding interface: when the interface will be released on client (either when it comes out of scope or set to {\f1\fs20 nil}), it will be released on the server side - a numerical identifier will be transmitted with all JSON requests
|{\f1\fs20 sicPerSession}|One object instance will be maintained during the whole running @*session@
|{\f1\fs20 sicPerUser}|One object instance will be maintained and associated with the running user
|{\f1\fs20 sicPerGroup}|One object instance will be maintained and associated with the running user's authorization group
|{\f1\fs20 sicPerThread}|One object instance will be maintained and associated with the running thread
|%
Of course, {\f1\fs20 sicPerSession}, {\f1\fs20 sicPerUser} and {\f1\fs20 sicPerGroup} modes will expect a specific user to be authenticated. Those implementation patterns will therefore only be available if the RESTful authentication is enabled between client and server.
Typical use of each mode may be the following:
|%24%76
|\b Lifetime|Use case\b0
|{\f1\fs20 sicSingle}|An asynchronous process (may be resource consuming)
|{\f1\fs20 sicShared}|Either a very simple process, or requiring some global data
|{\f1\fs20 sicClientDriven}|The best candidate to implement a Business Logic workflow
|{\f1\fs20 sicPerSession}|To maintain some data specific to the client application
|{\f1\fs20 sicPerUser}|Access to some data specific to one user
|{\f1\fs20 sicPerGroup}|Access to some data shared by a user category (e.g. administrators, or guests)
|{\f1\fs20 sicPerThread}|Thread-oriented process (e.g. for proper library initialization)
|%
In the current implementation of the framework, the class instance is allocated in memory.
This has two consequences:
- In client-server architecture, it is very likely that a lot of such instances will be created. It is therefore mandatory that it won't consume a lot of resource, especially with long-term life time: e.g. you should not store any BLOB within these instances, but try to restrict the memory use to the minimum. For a more consuming operation (a process which may need memory and CPU power), the {\f1\fs20 sicSingle} mode is preferred.
- There is no built-in data durability yet: service implementation shall ensure that data remaining in memory between calls (i.e. when not defined in {\f1\fs20 sicSingle} mode) won't be missing in case of server shutdown. It is up to the class to persist the needed data - using e.g. @3@.
Note also that all those life-time modes expect the method implementation code to be {\i thread-safe} and reintrant on the server side - only exceptions are {\f1\fs20 sicSingle} mode, which will have its own running instance, and {\f1\fs20 sicPerThread}, which will have its methods always run in the same thread context. In practice, the same user can open more than one connection, therefore it is recommended to protect all implementation class method process, or set the execution options as expected - see @72@.
In order to illustrate {\f1\fs20 sicClientDriven} implementation mode, let's introduce the following interface and its implementation (extracted from the supplied regression tests of the framework):
!type
!  IComplexNumber = interface(IInvokable)
!    ['{29D753B2-E7EF-41B3-B7C3-827FEB082DC1}']
!    procedure Assign(aReal, aImaginary: double);
!    function GetImaginary: double;
!    function GetReal: double;
!    procedure SetImaginary(const Value: double);
!    procedure SetReal(const Value: double);
!    procedure Add(aReal, aImaginary: double);
!    property Real: double read GetReal write SetReal;
!    property Imaginary: double read GetImaginary write SetImaginary;
!  end;
Purpose of this interface is to store a complex number within its internal fields, then retrieve their values, and define a "{\f1\fs20 Add}" method, to perform an addition operation. We used properties, with associated getter and setter methods, to provide object-like behavior on {\f1\fs20 Real} and {\f1\fs20 Imaginary} fields, in the code.
This interface is implemented on the server side by the following class:
!type
!  TServiceComplexNumber = class(TInterfacedObject,IComplexNumber)
!  private
!    fReal: double;
!    fImaginary: double;
!    function GetImaginary: double;
!    function GetReal: double;
!    procedure SetImaginary(const Value: double);
!    procedure SetReal(const Value: double);
!  public
!    procedure Assign(aReal, aImaginary: double);
!    procedure Add(aReal, aImaginary: double);
!    property Real: double read GetReal write SetReal;
!    property Imaginary: double read GetImaginary write SetImaginary;
!  end;
!
!{ TServiceComplexNumber }
!
!procedure TServiceComplexNumber.Add(aReal, aImaginary: double);
!begin
!  fReal := fReal+aReal;
!  fImaginary := fImaginary+aImaginary;
!end;
!
!procedure TServiceComplexNumber.Assign(aReal, aImaginary: double);
!begin
!  fReal := aReal;
!  fImaginary := aImaginary;
!end;
!
!function TServiceComplexNumber.GetImaginary: double;
!begin
!  result := fImaginary;
!end;
!
!function TServiceComplexNumber.GetReal: double;
!begin
!  result := fReal;
!end;
!
!procedure TServiceComplexNumber.SetImaginary(const Value: double);
!begin
!  fImaginary := Value;
!end;
!
!procedure TServiceComplexNumber.SetReal(const Value: double);
!begin
!  fReal := Value;
!end;
This interface is registered on the server side as such:
! Server.ServiceDefine(TServiceComplexNumber,[IComplexNumber],sicClientDriven);
Using the {\f1\fs20 sicClientDriven} mode, also the client side will be able to have its own life time handled as expected. That is, both {\f1\fs20 fReal} and {\f1\fs20 fImaginary} field will remain allocated on the server side as long as needed. A time-out driven @*garbage collector@ will delete any un-closed pending session, therefore release resources allocted in {\f1\fs20 sicClientDriven} mode, even in case of a broken connection.
:107  Accessing low-level execution context
:   Retrieve information from the global ServiceContext
When any {\f1\fs20 interface}-based service is executed, a global {\f1\fs20 threadvar} named {\f1\fs20 @**ServiceContext@} can be accessed to retrieve the currently running context on the server side.
You will have access to the following information, which could be useful for {\f1\fs20 sicPerSession, sicPerUser} and {\f1\fs20 sicPerGroup} instance life time modes:
!  TServiceRunningContext = record
!    /// the currently running service factory
!    // - it can be used within server-side implementation to retrieve the
!    // associated TSQLRestServer instance
!    // - note that TServiceFactoryServer.Get() won't override this value, when
!    // called within another service (i.e. if Factory is not nil)
!    Factory: TServiceFactoryServer;
!    /// the currently runnning context which launched the method
!    // - low-level RESTful context is also available in its Call member
!    // - Request.Server is the safe access point to the underlying TSQLRestServer,
!    // unless the service is implemented via TInjectableObjectRest, so the
!    // TInjectableObjectRest.Server property is preferred
!    // - make available e.g. current session or authentication parameters
!    // (including e.g. user details via Request.Server.SessionGetUser)
!    Request: TSQLRestServerURIContext;
!    /// the thread which launched the request
!    // - is set by TSQLRestServer.BeginCurrentThread from multi-thread server
!    // handlers - e.g. TSQLite3HttpServer or TSQLRestServerNamedPipeResponse
!    RunningThread: TThread;
!  end;
When used, a local copy or a {\f1\fs20 PServiceRunningContext} pointer should better be created, since accessing a {\f1\fs20 threadvar} has a non negligible performance cost.
If your code is compiled within some @*packages@, {\f1\fs20 threadvar} read won't work, due to a {\i Delphi} compiler/RTL restriction (bug?). In such case, you have to call the following {\f1\fs20 function} instead of directly access the {\f1\fs20 threadvar}:
! function CurrentServiceContext: TServiceRunningContext;
Note that this global {\f1\fs20 threadvar} is reset to 0 outside an {\f1\fs20 interface}-based service method call. It will therefore be useless to read it from a method-based service, for instance.
:185   Implement your service from TInjectableObjectRest
An issue with the {\f1\fs20 @*ServiceContext@} threadvar is that the execution context won't be filled when a SOA method is executed outside a client/server context, e.g. if the {\f1\fs20 TSQLRestServer} instance did resolve itself its dependencies using {\f1\fs20 Services.Resolve()}.
A safer (and slightly faster) alternative is to implement your service by inheriting from the {\f1\fs20 @**TInjectableObjectRest@} class.\line This {\f1\fs20 class} has its own {\f1\fs20 Resolve()} overloaded methods (inherited from {\f1\fs20 TInjectableObject}), but also two additional properties:
!  TInjectableObjectRest = class(TInjectableObject)
!  ...
!  public
!    property Factory: TServiceFactoryServer read fFactory;
!    property Server: TSQLRestServer read fServer;
!  end;
Those properties will be injected by {\f1\fs20 TServiceFactoryServer.CreateInstance}, i.e. when the service implementation object will be instantiated on the server side. They will give direct and safe access to the underlying REST server, e.g. all its @*ORM@ methods.
:  Using services on the Server side
Once the service is registered on the server side, it is very easy to use it in your code.
In a complex @17@, it is not a good practice to have services calling each other. Code decoupling is a key to maintainability here. But in some cases, you'll have to consume services on the server side, especially if your software architecture has several layers (like in a @54@): your application services could be decoupled, but the {\i @*Domain-Driven@} services (those implementing the business model) could be on another Client-Server level, with a dedicated protocol, and could have nested calls.
In this case, according to the @47@, you'd better rely on abstraction in your code, i.e. not call the service implementation (i.e. the {\f1\fs20 TInterfacedObject} instances or even worse directly the low-level classes or functions), but the service abstract {\f1\fs20 interface}. You can use the following method of your {\f1\fs20 TSQLRest.Services} instance (note that this method is available on both client and server sides as abstract {\f1\fs20 @**TServiceFactory@}, so is the right access point to all services):
! function TServiceFactory.Get(out Obj): Boolean;
You have several methods to retrieve a {\f1\fs20 TServiceFactory} instance, either from the service name, its @*GUID@, or its index in the list.
That is, you may code:
!var I: ICalculator;
!begin
!  if ServiceContext.Request.Server.Services['Calculator'].Get(I)) then
!    result := I.Add(10,20);
!end;
or, for a more complex service:
!var CN: IComplexNumber;
!begin
!  if not ServiceContext.Request.Server.Services.Resolve(IComplexNumber,CN) then
!    exit; // IComplexNumber interface not found
!  CN.Real := 0.01;
!  CN.Imaginary := 3.1415;
!  CN.Add(100,200);
!  assert(SameValue(CN.Real,100.01));
!  assert(SameValue(CN.Imaginary,203.1415));
!end; // here CN will be released
For newer generic-aware versions of {\i Delphi} (i.e. {\i Delphi} 2010 and up, since {\i Delphi} 2009 is buggy about generics), you can use such a method, which enables compile-time checking:
!var I: ICalculator;
!begin
!  I := Server.Service<ICalculator>;
!  if I<>nil then
!    result := I.Add(10,20);
!end;
You can of course cache/store your {\f1\fs20 TServiceFactory} or {\f1\fs20 TSQLRest} instances within a local field, if you wish. Using {\f1\fs20 ServiceContext.Request.Server} is verbose and error-prone.\line But you may consider instead to @185@: the {\f1\fs20 @*TInjectableObjectRest@} class has already its buil-in {\f1\fs20 Resolve()} overloaded methods, and direct access to the underlying {\f1\fs20 Server: TSQLRestServer} instance. So you will be able to write directly both SOA and ORM code:
!var I: ICalculator;
!begin
!  if Resolve(ICalculator,I) then
!    Server.Add(TSQLRecordExecution,['Add',I.Add(10,20)]);
!end;
If the service has been defined as {\f1\fs20 sicPerThread}, the instance you will retrieve on the server side will also be specific to the running thread - in this case, caching the instance may be source of confusion, since there will be one dedicated instance per thread.
\page
: Client side
There is no implementation at all on the client side. This is the magic of {\i mORMot}'s services: no Wizard to call (as in {\i DataSnap}, {\i RemObjects} or {\i WCF}), nor client-side methods to write - as with our @49@.
You just register the existing {\f1\fs20 interface} definition (e.g. our {\f1\fs20 ICalculator} type), and you can remotely access to all its methods, executed on the server side.
In fact, a hidden "fake" {\f1\fs20 TInterfaceObject} class will be created by the framework (including its internal {\i VTable} and low-level assembler code), and used to interact with the remote server. But you do not have to worry about this process: it is transparent to your code.
:  Set up the Client factory
On the client side, you have to register the corresponding interface to initialize its associated @*factory@, as such:
! Client.ServiceRegister([TypeInfo(ICalculator)],sicShared);
You may prefer @141@, if the type has previously been registered:
! Client.ServiceDefine([ICalculator],sicShared);
It is very close to the Server-side registration, despite the fact that we do not provide any implementation class here. Implementation will remain on the server side.
Note that the implementation mode (here {\f1\fs20 sicShared}) shall match the one used on the server side. An error will occur if this setting is not coherent.
The other interface we talked about, i.e. {\f1\fs20 IComplexNumber}, is registered as such for the client:
! Client.ServiceDefine([IComplexNumber],sicClientDriven);
This will create the corresponding {\f1\fs20 TServiceFactoryClient} instance, ready to serve fake implementation classes to the client process.
To be more precise, this registration step is indeed not mandatory on the client side. If you use the {\f1\fs20 TServiceContainerClient.Info()} method, the client-side implementation will auto-register the supplied interface, in {\f1\fs20 sicClientDriven} implementation mode.
:  Using services on the Client side
Once the service is registered on the client side, it is very easy to use it in your code.
You can use the same methods as on the server side to retrieve a {\f1\fs20 TServiceFactory} instance.
That is, you may code:
!var I: ICalculator;
!begin
!  if Client.Services['Calculator'].Get(I)) then
!    result := I.Add(10,20);
!end;
For {\i Delphi} 2010 and up, you can use a generic-based method, which enables compile-time checking:
!var I: ICalculator;
!begin
!  I := Client.Service<ICalculator>;
!  if I<>nil then
!    result := I.Add(10,20);
!end;
For a more complex service, initialized as {\f1\fs20 sicClientDriven}:
!var CN: IComplexNumber;
!begin
!  if not Client.Services.Resolve(IComplexNumber,CN) then
!    exit; // IComplexNumber interface not found
!  CN.Real := 0.01;
!  CN.Imaginary := 3.1415;
!  CN.Add(100,200);
!  assert(SameValue(CN.Real,100.01));
!  assert(SameValue(CN.Imaginary,203.1415));
!end; // here CN will be released on both client AND SERVER sides
The code is just the same as on the server. The only functional change is that the execution will take place on the server side (using the registered {\f1\fs20 TServiceComplexNumber} implementation class), and the corresponding class instance will remain active until the {\f1\fs20 CN} local interface will be released on the client.
You can of course cache your {\f1\fs20 TServiceFactory} instance within a local field, if you wish. On the client side, even if the service has been defined as {\f1\fs20 sicPerThread}, you can safely cache and reuse the same instance, since the {\i per-thread} process will take place on the server side only.
As we stated in the previous paragraph, since the {\f1\fs20 IComplexNumber} is to be executed as {\f1\fs20 sicClientDriven}, it is not mandatory to call the {\f1\fs20 Client.ServiceRegister} or {\f1\fs20 ServiceDefine} method for this interface. In fact, during {\f1\fs20 Client.Services.Info(TypeInfo(IComplexNumber))} method execution, the registration will take place, if it has not been done explicitly before. For code readability, it may be a good idea to explicitly register the interface on the client side also, just to emphasize that this interface is about to be used, and in which mode.
\page
:89 Sample code
You can find in the "{\f1\fs20 SQLite3/Samples/14 - Interface based services}" folder of the supplied source code distribution, a dedicated sample about this feature.
Purpose of this code is to show how to create a client-server service, using interfaces, over named pipe communication.
:  The shared contract
First, you'll find a common unit, shared by both client and server applications:
!unit Project14Interface;
!
!interface
!
!type
!  ICalculator = interface(IInvokable)
!    ['{9A60C8ED-CEB2-4E09-87D4-4A16F496E5FE}']
!    function Add(n1,n2: integer): integer;
!  end;
!
!const
!  ROOT_NAME = 'service';
!  PORT_NAME = '888';
!  APPLICATION_NAME = 'RestService';
!
!implementation
!
!uses mORMot;
!
!initialization
!  TInterfaceFactory.RegisterInterfaces([TypeInfo(ICalculator)]);
!end.
Unique purpose of this unit is to define the service {\f1\fs20 interface}, and the {\f1\fs20 ROOT_NAME} used for the ORM Model (and therefore RESTful URI scheme), and the {\f1\fs20 APPLICATION_NAME} used for named-pipe communication.
This {\f1\fs20 ICalculator} type is also registered for the internal {\f1\fs20 interface} factory system, so that you could use the framework methods directly with {\f1\fs20 ICalculator} instead of {\f1\fs20 TypeInfo(ICalculator)}.
:  The server sample application
The server is implemented as such:
!program Project14Server;
!
!{$APPTYPE CONSOLE}
!
!uses
!  SysUtils,
!  mORMot,
!  mORMotSQLite3,
!  Project14Interface;
!
!type
!  TServiceCalculator = class(TInterfacedObject, ICalculator)
!  public
!    function Add(n1,n2: integer): integer;
!  end;
!
!function TServiceCalculator.Add(n1, n2: integer): integer;
!begin
!  result := n1+n2;
!end;
!
!var
!  aModel: TSQLModel;
!begin
!  aModel := TSQLModel.Create([],ROOT_NAME);
!  try
!    with TSQLRestServerDB.Create(aModel,ChangeFileExt(paramstr(0),'.db'),true) do
!    try
!      CreateMissingTables; // we need AuthGroup and AuthUser tables
!!      ServiceDefine(TServiceCalculator,[ICalculator],sicShared);
!      if ExportServerNamedPipe(APPLICATION_NAME) then
!        writeln('Background server is running.'#10) else
!        writeln('Error launching the server'#10);
!      write('Press [Enter] to close the server.');
!      readln;
!    finally
!      Free;
!    end;
!  finally
!    aModel.Free;
!  end;
!end.
It will instantiate a {\f1\fs20 @*TSQLRestServerDB@} class, containing a {\i @*SQLite3@} database engine. In fact, since we need authentication, both {\f1\fs20 AuthGroup} and {\f1\fs20 AuthUser} tables are expected to be available.
Then a call to {\f1\fs20 ServiceDefine()} will define the {\f1\fs20 ICalculator} contract, and the {\f1\fs20 TServiceCalculator} class to be used as its implementation. The {\f1\fs20 sicShared} mode is used, since the same implementation class can be shared during all calls (there is no shared nor private data to take care).
Note that since the database expectations of this server are basic (only CRUD commands are needed to handle authentication tables), we may use a {\f1\fs20 @*TSQLRestServerFullMemory@} class instead of {\f1\fs20 TSQLRestServerDB}. This is what is the purpose of the {\f1\fs20 Project14ServerInMemory.dpr} sample:
!program Project14ServerInMemory;
!  (...)
!!    with TSQLRestServerFullMemory.Create(aModel,'test.json',false,true) do
!    try
!!      ServiceDefine(TServiceCalculator,[ICalculator],sicShared);
!      if ExportServerNamedPipe(APPLICATION_NAME) then
!  (...)
Using this class will include the {\f1\fs20 CreateMissingTables} call to create both {\f1\fs20 AuthGroup} and {\f1\fs20 AuthUser} tables needed for authentication. But the resulting executable will be lighter: only 200 KB when compiled with {\i Delphi} 7 and our LVCL classes, for a full service provider.
:  The client sample application
The client is just a simple form with two {\f1\fs20 TEdit} fields ({\f1\fs20 edtA} and {\f1\fs20 edtB}), and a "{\i Call}" button, which {\f1\fs20 OnClick} event is implemented as:
!procedure TForm1.btnCallClick(Sender: TObject);
!var a,b: integer;
!    err: integer;
!    I: ICalculator;
!begin
!  val(edtA.Text,a,err);
!  if err<>0 then begin
!    edtA.SetFocus;
!    exit;
!  end;
!  val(edtB.Text,b,err);
!  if err<>0 then begin
!    edtB.SetFocus;
!    exit;
!  end;
!  if Client=nil then begin
!    if Model=nil then
!!      Model := TSQLModel.Create([],ROOT_NAME);
!!    Client := TSQLRestClientURINamedPipe.Create(Model,APPLICATION_NAME);
!!    Client.SetUser('User','synopse');
!!    Client.ServiceDefine([ICalculator],sicShared);
!  end;
!!  if Client.Services['Calculator'].Get(I) then
!!    lblResult.Caption := IntToStr(I.Add(a,b));
!end; // here local I will be released
The client code is initialized as such:
- A {\f1\fs20 TSQLRestClientURINamedPipe} instance is created, with an associate {\f1\fs20 TSQLModel} and the given {\f1\fs20 APPLICATION_NAME} to access the proper server via a named pipe communication;
- The connection is authenticated with the default {\f1\fs20 'User'} rights;
- The {\f1\fs20 ICalculator} interface is defined in the client's internal @*factory@, in {\f1\fs20 sicShared} mode (just as in the server).
Once the client is up and ready, the local {\f1\fs20 I: ICalculator} variable instance is retrieved, and the remote service is called directly via a simple {\f1\fs20 I.Add(a,b)} statement.
You can imagine how easy and safe it will be to implement a @17@ for your future applications, using {\i mORMot}.
:74  Enhanced sample: remote SQL access
You will find in the {\f1\fs20 SQLite3\\Samples\\16 - Execute SQL via services} folder of {\i mORMot} source code a @*Client-Server@ sample able to access any external database via @*JSON@ and HTTP. It is a good demonstration of how to use a non-trivial @*interface@-based service between a client and a server. It will also show how our {\f1\fs20 SynDB.pas} classes have a quite abstract design, and are easy to work with, whatever database provider you need to use.
The corresponding service contract has been defined:
!  TRemoteSQLEngine = (rseOleDB, rseODBC, rseOracle, rseSQlite3, rseJet, rseMSSQL);
!
!  IRemoteSQL = interface(IInvokable)
!    ['{9A60C8ED-CEB2-4E09-87D4-4A16F496E5FE}']
!    procedure Connect(aEngine: TRemoteSQLEngine; const aServerName, aDatabaseName,
!      aUserID, aPassWord: RawUTF8);
!    function GetTableNames: TRawUTF8DynArray;
!    function Execute(const aSQL: RawUTF8; aExpectResults, aExpanded: Boolean): RawJSON;
!  end;
Purpose of this service is:
- To {\f1\fs20 Connect()} to any external database, given the parameters as expected by a standard {\f1\fs20 TSQLDBConnectionProperties.Create()} constructor call;
- Retrieve all table names of this external database as a list;
- Execute any SQL statement, returning the content as JSON array, ready to be consumed by AJAX applications (if {\f1\fs20 aExpanded} is {\f1\fs20 true}), or a {\i Delphi} client (e.g. via a {\f1\fs20 TSQLTableJSON} and the {\f1\fs20 mORMotUI} unit).
Of course, this service will be defined in {\f1\fs20 sicClientDriven} mode. That is, the framework will be able to manage a client-driven {\f1\fs20 TSQLDBProperties} instance life time.
Benefit of this service is that no database connection is required on the client side: a regular HTTP connection is enough. No need to neither install nor configure any database provider.
Due to {\i mORMot} optimized JSON serialization, it will probably be faster to work with such plain HTTP / JSON services, instead of a database connection through a VPN. In fact, database connections are made to work on a local network, and do not like high-latency connections, which are typical on the Internet. On the contrary, the {\i mORMot} Client-Server process is optimized for such kind of connection.
Note that the {\f1\fs20 Execute()} method returns a {\f1\fs20 RawJSON} kind of variable, which is in fact a sub-type of {\f1\fs20 RawUTF8}. Its purpose is to transmit the @*UTF-8@ encoded content directly, with no translation to a JSON string, as will be the case with a {\f1\fs20 RawUTF8} variable. In fact, escaping some JSON array within a JSON string is quite verbose. Using {\f1\fs20 RawJSON} in this case ensure the best client-side and server-side speed, and also reduce the transmission bandwidth.
The server part is quite easy to follow:
!type
!  TServiceRemoteSQL = class(TInterfacedObject, IRemoteSQL)
!  protected
!    fProps: TSQLDBConnectionProperties;
!  public
!    destructor Destroy; override;
!  public // implements IRemoteSQL methods
!    procedure Connect(aEngine: TRemoteSQLEngine; const aServerName, aDatabaseName,
!      aUserID, aPassWord: RawUTF8);
!    function GetTableNames: TRawUTF8DynArray;
!    function Execute(const aSQL: RawUTF8; aExpectResults, aExpanded: Boolean): RawJSON;
!  end;
!
!{ TServiceRemoteSQL }
!
!procedure TServiceRemoteSQL.Connect(aEngine: TRemoteSQLEngine;
!  const aServerName, aDatabaseName, aUserID, aPassWord: RawUTF8);
!const // rseOleDB, rseODBC, rseOracle, rseSQlite3, rseJet, rseMSSQL
!  TYPES: array[TRemoteSQLEngine] of TSQLDBConnectionPropertiesClass = (
!     TOleDBConnectionProperties, TODBCConnectionProperties,
!     TSQLDBOracleConnectionProperties, TSQLDBSQLite3ConnectionProperties,
!     TOleDBJetConnectionProperties, TOleDBMSSQL2008ConnectionProperties);
!begin
!  if fProps<>nil then
!    raise Exception.Create('Connect called more than once');
!  fProps := TYPES[aEngine].Create(aServerName,aDatabaseName,aUserID,aPassWord);
!end;
!
!function TServiceRemoteSQL.Execute(const aSQL: RawUTF8; aExpectResults, aExpanded: Boolean): RawJSON;
!var res: ISQLDBRows;
!begin
!  if fProps=nil then
!    raise Exception.Create('Connect call required before Execute');
!  res := fProps.ExecuteInlined(aSQL,aExpectResults);
!  if res=nil then
!    result := '' else
!    result := res.FetchAllAsJSON(aExpanded);
!end;
!
!function TServiceRemoteSQL.GetTableNames: TRawUTF8DynArray;
!begin
!  if fProps=nil then
!    raise Exception.Create('Connect call required before GetTableNames');
!  fProps.GetTableNames(result);
!end;
!
!destructor TServiceRemoteSQL.Destroy;
!begin
!  FreeAndNil(fProps);
!  inherited;
!end;
Any exception during {\f1\fs20 SynDB.pas} process, or raised manually in case of wrong use case will be transmitted to the client, just as expected. The {\f1\fs20 fProps} instance life-time is handled by the client, so all we need is to release its pointer in the service implementation {\f1\fs20 destructor}.
The services are initialized on the server side with the following code:
!var
!  aModel: TSQLModel;
!  aServer: TSQLRestServer;
!  aHTTPServer: TSQLHttpServer;
!begin
!  // define the log level
!  with TSQLLog.Family do begin
!    Level := LOG_VERBOSE;
!    EchoToConsole := LOG_VERBOSE; // log all events to the console
!  end;
!  // manual switch to console mode
!  AllocConsole;
!  TextColor(ccLightGray);
!  // create a Data Model
!  aModel := TSQLModel.Create([],ROOT_NAME);
!  try
!    // initialize a TObjectList-based database engine
!    aServer := TSQLRestServerFullMemory.Create(aModel,'users.json',false,true);
!!    try
!      // register our IRemoteSQL service on the server side
!!      aServer.ServiceRegister(TServiceRemoteSQL,[TypeInfo(IRemoteSQL)],sicClientDriven).
!        // fProps should better be executed and released in the one main thread
!!        SetOptions([],[optExecInMainThread,optFreeInMainThread]);
!      // launch the HTTP server
!      aHTTPServer := TSQLHttpServer.Create('888',[aServer],'+',useHttpApiRegisteringURI);
!      try
!        aHTTPServer.AccessControlAllowOrigin := '*'; // for AJAX requests to work
!        writeln(#10'Background server is running.'#10);
!        writeln('Press [Enter] to close the server.'#10);
!        ConsoleWaitForEnterKey;
!      finally
!        aHTTPServer.Free;
!      end;
!    finally
!      aServer.Free;
!    end;
!  finally
!    aModel.Free;
!  end;
!end.
This is a typical {\i mORMot} server initialization, published over the HTTP communication protocol (with auto-registration feature, if possible, as stated by the {\f1\fs20 useHttpApiRegisteringURI} flag). Since we won't use ORM for any purpose but authentication, a fast {\f1\fs20 TObjectList}-based engine (i.e. {\f1\fs20 TSQLRestServerFullMemory}) is enough for this sample purpose.
In the above code, you can note that {\f1\fs20 IRemoteSQL} service is defined with the {\f1\fs20 optExecInMainThread} and {\f1\fs20 optFreeInMainThread} options. It means that all methods will be executed in the main process thread. In practice, since {\f1\fs20 SynDB.pas} database access may open one connection per thread (e.g. for {\i @*OleDB@} / {\i @*MS SQL@} or {\i @*Oracle@} providers), it may use a lot of memory. Forcing the database execution in the main thread will lower the resource consumption, and still will perform with decent speed (since all the internal marshalling and communication will be multi-threaded in the framework units).
From the client point of view, it will be consumed as such:
!procedure TMainForm.FormShow(Sender: TObject);
!  (...)
!  fModel := TSQLModel.Create([],ROOT_NAME);
!!  fClient := TSQLHttpClient.Create('localhost','888',fModel);
!  if not fClient.ServerTimestampSynchronize then begin
!    ShowLastClientError(fClient,'Please run Project16ServerHttp.exe');
!    Close;
!    exit;
!  end;
!!  if (not fClient.SetUser('User','synopse')) or
!!     (not fClient.ServiceRegisterClientDriven(TypeInfo(IRemoteSQL),fService)) then begin
!    ShowLastClientError(fClient,'Remote service not available on server');
!    Close;
!    exit;
!  end;
!end;
Our {\f1\fs20 IRemoteSQL} service will be accessed in {\f1\fs20 sicClientDriven} mode, so here we need to initialize @*REST@ful authentication - see @18@ - with a proper call to {\f1\fs20 SetUser()}.
Note the use of {\f1\fs20 ShowLastClientError()} function of {\f1\fs20 mORMotUILogin} unit, which is able to use our {\f1\fs20 SynTaskDialog} unit to report standard and detailed information about the latest error.
In this sample, no table has been defined within the ORM model. It is not necessary, since all external process will take place at the SQL level. As we need authentication (see the call to {\f1\fs20 fClient.SetUser} method), the ORM core will by itself add the {\f1\fs20 @*TSQLAuthUser@} and {\f1\fs20 @*TSQLAuthGroup@} tables to the model - no need to add them explicitly.
From now on, we have a {\f1\fs20 fService: IRemoteSQL} instance available to connect and process any remote SQL request.
!procedure TMainForm.btnOpenClick(Sender: TObject);
!var TableNames: TRawUTF8DynArray;
!  (...)
!  with fSettings do
!!    fService.Connect(Engine,ServerName,DatabaseName,UserID,PassWord);
!!  TableNames := fService.GetTableNames;
!  cbbTableNames.Items.Text := UTF8ToString(RawUTF8ArrayToCSV(TableNames,#13#10));
!  (...)
Now we are connected to the database via the remote service, and we retrieved the table names in a {\f1\fs20 TComboBox}.
Then a particular SQL statement can be executed as such:
!procedure TMainForm.btnExecuteClick(Sender: TObject);
!var SQL: RawUTF8;
!begin
!  SQL := trim(StringToUTF8(mmoQuery.Text));
!  Screen.Cursor := crHourGlass;
!  try
!    try
!      if isSelect(pointer(SQL)) then begin
!!        fTableJSON := fService.Execute(SQL,True,False);
!        TSQLTableToGrid.Create(drwgrdData,
!          TSQLTableJSON.Create([],SQL,pointer(fTableJSON),Length(fTableJSON)),fClient);
!      end else
!        fService.Execute(SQL,False,False);
!    except
!      on E: Exception do
!        ShowException(E);
!    end;
!  finally
!    Screen.Cursor := crDefault;
!  end;
!end;
Here, {\f1\fs20 TSQLTableToGrid.Create()}, from the {\f1\fs20 mORMotUI} unit, will "inject" the returned data to a standard {\f1\fs20 TDrawGrid}, using a {\f1\fs20 TSQLTableJSON} instance to un-serialize the returned JSON content.
Note that in case of any exception (connection failure, or server side error, e.g. wrong SQL statement), the {\f1\fs20 ShowExecption()} method is used to notify the user with appropriate information.
\page
:149 Asynchronous callbacks
When publishing @*SOA@ services, most of them are defined as {\i @*stateless@}, in a typical query/answer pattern - see @17@. This fits exactly with the {\i @*RESTful@} approach of @63@, as proposed by the framework.
But it may happen that a client application (or service) needs to know the state of a given service. In a pure {\i stateless} implementation, it will have to {\i query} the server for any state change, i.e. for any pending notification - this is called {\i polling}.
{\i Polling} may take place for instance:
- When a time consuming work is to be processed on the server side. In this case, the client could not wait for it to be finished, without raising a timeout on the HTTP connection: as a workaround, the client may start the work, then ask for its progress status regularly using a timer and a dedicated method call;
- When an unpredictable event is to be notified from the server side. In this case, the client should ask regularly (using a timer, e.g. every second), for any pending event, then react on purpose.
It may therefore sounds preferred, and in some case necessary, to have the ability to let the server {\i notify} one or several clients without any prior query, nor having the requirement of a client-side timer:
- {\i Polling} may be pretty resource consuming on both client and server sides, and add some unwanted latency;
- If immediate notification is needed, some kind of "long polling" algorithm may take place, i.e. the server will wait for a long time before returning the notification state if no event did happen: in this case, a dedicated connection is required, in addition to the REST one;
- In an @*event-driven@ systems, a lot of messages are sent to the clients: a proper @*publish/subscribe@ mechanism is preferred, otherwise the complexity of polling methods may increase and become inefficient and unmaintainable;
- Explicit push notifications may be necessary, e.g. when a lot of potential events, associated with a complex set of parameters, are likely to be sent by the client.
Our {\i mORMot} framework is therefore able to easily implement asynchronous callbacks over {\i @*WebSockets@}, defining the callbacks as {\f1\fs20 interface} parameters in service method definitions - see @154@.
:150  WebSockets support
By definition, HTTP connections are stateless and one-way, i.e. a client sends a request to the server, which replies back with an answer. There is no way to let the server send a message to the client, without a prior request from the client side.
{\i @**WebSockets@} is a communication protocol which is able to {\i upgrade} a regular HTTP connection into a dual-way communication wire. After a safe handshake, the underlying TCP/IP socket is able to be accessed directly, via a set of lightweight {\i frames} over an application-defined {\i protocol}, without the HTTP overhead.
The {\f1\fs20 SynBidirSock.pas} unit implements low-level server and client {\i WebSockets} communication.
The {\f1\fs20 TWebSocketProtocol} class defines an abstract {\i WebSockets protocol}, currently implemented as several classes:
\graph HierTWebSocketProtocolJSON TWebSocketProtocolJSON classes hierarchy
\TWebSocketProtocolChat\TWebSocketProtocol
\TWebSocketProtocolRest\TWebSocketProtocol
\TWebSocketProtocolBinary\TWebSocketProtocolRest
\TWebSocketProtocolJSON\TWebSocketProtocolRest
\
For our @63@, we will still need to make {\i RESTful} requests, so the basic {\i WebSockets} framing has been enhanced to support {\f1\fs20 TWebSocketProtocolRest} REST-compatible protocols, able to use the single connection for both REST queries and asynchronous notifications.\line Two classes are available for your @*SOA@ applications:
- {\f1\fs20 TWebSocketProtocolJSON} as a "pure" JSON light protocol;
- {\f1\fs20 TWebSocketProtocolBinary} as a binary proprietary protocol, with optional frame compression and @*AES@ @*encryption@ (using @*AES-NI@ hardware instructions, if available).
In practice, on the server side, you will start your {\f1\fs20 TSQLHttpServer} by specifying {\f1\fs20 useBidirSocket} as kind of server:
! HttpServer := TSQLHttpServer.Create('8888',[Server],'+',useBidirSocket);
Under the hood, it will instantiate a {\f1\fs20 TWebSocketServer} HTTP server, as defined in {\f1\fs20 mORMotHttpServer.pas}, based on the sockets API, able to upgrade the HTTP protocol into {\i WebSockets}. Our @88@ is not yet able to switch to {\i WebSockets} - and at API level, it will require at least {\i Windows 8} or {\i Windows 2012 Server}.
Then you enable {\i WebSockets} for the {\f1\fs20 TWebSocketProtocolBinary} protocol, with a symmetric encryption key:
!   HttpServer.WebSocketsEnable(Server,'encryptionkey');
On the client side, you will use a {\f1\fs20 TSQLHttpClientWebsockets} instance, as defined in {\f1\fs20 mORMotHttpClient.pas}, then explicitly upgrade the connection to use {\i WebSockets} (since by default, it will stick to the HTTP protocol):
!  Client := TSQLHttpClientWebsockets.Create('127.0.0.1','8888',TSQLModel.Create([]));
!  Client.WebSocketsUpgrade('encryptionkey');
The expected protocol detail should match the one on the server, i.e. {\f1\fs20 'encryptionkey'} encryption over our binary protocol.
Once upgraded to {\i WebSockets}, you may use regular REST commands, as usual:
!  Client.ServerTimestampSynchronize;
But in addition to regular query/answer commands as defined for @63@, you will be able to define callbacks using {\f1\fs20 interface} parameters to the service methods.
Under the hood, both client and server will communicate using {\i WebSockets} frames, maintaining the connection active using heartbeats (via ping/pong frames), and with clean connection shutdown, from any side. You can use the {\f1\fs20 Settings} property of the {\f1\fs20 TWebSocketServerRest} instance, as returned by {\f1\fs20 TSQLHttpServer.WebSocketsEnable()}, to customize the low-level {\i WebSockets} protocol (e.g. timeouts or heartbeats) on the server side. The {\f1\fs20 TSQLHttpClientWebsockets.WebSockets}.{\f1\fs20 Settings} property will allow the same, on the client side.
We have observed, from our regression tests and internal benchmarking, that using our {\i WebSockets} may be faster than regular HTTP, since its frames will be sent as once, whereas HTTP headers and body are not sent in the same TCP packet, and compression will be available for the whole frame, whereas HTTP headers are not compressed. The ability to use strong AES encryption will make this mean of communication even safer than plain HTTP, even with @151@.
:152   Using a "Saga" callback to notify long term end-of-process
An example is better than 100 talks.\line So let's take a look at the {\f1\fs20 Project31LongWorkServer.dpr} and {\f1\fs20 Project31LongWorkClient.dpr} samples, from the {\f1\fs20 SQLite3\\Samples\\31 - WebSockets} sub-folder. They will implement a client/server application, in which the client launches a long term process on the server side, then is notified when the process is done, either with success, or failure.\line Such a pattern is very common in the @*SOA@ world, and also known as "saga" - see @http://www.rgoarchitects.com/Files/SOAPatterns/Saga.pdf - but in practice, it may be difficult to implement it safely and easily. Let's see how our framework make writing sagas a breeze.
First we define the {\f1\fs20 interface}s to be used, in a shared {\f1\fs20 Project31LongWorkCallbackInterface.pas} unit:
!type
!  ILongWorkCallback = interface(IInvokable)
!    ['{425BF199-19C7-4B2B-B1A4-A5BE7A9A4748}']
!    procedure WorkFinished(const workName: string; timeTaken: integer);
!    procedure WorkFailed(const workName, error: string);
!  end;
!
!  ILongWorkService = interface(IInvokable)
!    ['{09FDFCEF-86E5-4077-80D8-661801A9224A}']
!    procedure StartWork(const workName: string; const onFinish: ILongWorkCallback);
!    function TotalWorkCount: Integer;
!  end;
The only specific definition is the {\f1\fs20 const onFinish: ILongWorkCallback} parameter, supplied to the {\f1\fs20 ILongWorkService.StartWork()} method. The client will create a class implementing {\f1\fs20 ILongWorkCallback}, then specify it as parameter to this method. On the server side, a "fake" class will implement {\f1\fs20 ILongWorkCallback}, then will call back the client using the very same {\i WebSockets} connection, when any of its methods will be executed.
As you can see, a single callback {\f1\fs20 interface} instance may have several methods, with their own set of parameters (here {\f1\fs20 WorkFinished} and {\f1\fs20 WorkFailed}), so that the callback may be quite expressive. Any kind of usual parameters will be transmitted, after serialization: {\f1\fs20 string}, {\f1\fs20 integer}, but even {\f1\fs20 record}, {\i dynamic arrays}, {\f1\fs20 @*TSQLRecord@} or {\f1\fs20 @*TPersistent@} values.
When the {\f1\fs20 ILongWorkCallback} instance will be released on the client side, the server will  be notified, so that any further notification won't create a connection error. We will see later how to handle those events.
:   Client service consumption
The client may be connected to the server as such (see the {\f1\fs20 Project31LongWorkClient.dpr} sample source code for the full details, including error handling):
!var Client: TSQLHttpClientWebsockets;
!    workName: string;
!    Service: ILongWorkService;
!    callback: ILongWorkCallback;
!begin
!  Client := TSQLHttpClientWebsockets.Create('127.0.0.1','8888',TSQLModel.Create([]));
!!  Client.WebSocketsUpgrade(PROJECT31_TRANSMISSION_KEY);
!  Client.ServiceDefine([ILongWorkService],sicShared);
!  Client.Services.Resolve(ILongWorkService,Service);
Then we define our callback, using a dedicated class:
!type
!  TLongWorkCallback = class(TInterfacedCallback,ILongWorkCallback)
!  protected
!    procedure WorkFinished(const workName: string; timeTaken: integer);
!    procedure WorkFailed(const workName, error: string);
!  end;
!
!procedure TLongWorkCallback.WorkFailed(const workName, error: string);
!begin
!  writeln(#13'Received callback WorkFailed(',workName,') with message "',error,'"');
!end;
!
!procedure TLongWorkCallback.WorkFinished(const workName: string;
!  timeTaken: integer);
!begin
!  writeln(#13'Received callback WorkFinished(',workName,') in ',timeTaken,'ms');
!end;
Then we specify this kind of callback as parameter to start a long term work:
!!    callback := TLongWorkCallback.Create(Client,ILongWorkCallback);
!    try
!      repeat
!        readln(workName);
!        if workName='' then
!          break;
!!        Service.StartWork(workName,callback);
!      until false;
!    finally
!      callback := nil; // the server will be notified and release its "fake" class
!      Service := nil;  // release the service local instance BEFORE Client.Free
!    end;
As you can see, the client is able to start one or several work processes, then expects to be notified of the process ending on its callback {\f1\fs20 interface} instance, without explicitly polling the server for its state, since the connection was upgraded to {\i WebSockets} via a call to {\f1\fs20 TSQLHttpClientWebsockets.WebSocketsUpgrade()}.
:   Server side implementation
The server will define the working thread as such (see the {\f1\fs20 Project31LongWorkServer.dpr} sample source code for the full details):
!type
!  TLongWorkServiceThread = class(TThread)
!  protected
!!    fCallback: ILongWorkCallback;
!    fWorkName: string;
!    procedure Execute; override;
!  public
!    constructor Create(const workName: string; const callback: ILongWorkCallback);
!  end;
!
!constructor TLongWorkServiceThread.Create(const workName: string;
!  const callback: ILongWorkCallback);
!begin
!  inherited Create(false);
!!  fCallback := Callback;
!  fWorkName := workName;
!  FreeOnTerminate := true;
!end;
!
!procedure TLongWorkServiceThread.Execute;
!var tix: Int64;
!begin
!  tix := GetTickCount64;
!!  Sleep(5000+Random(1000)); // some hard work
!  if Random(100)>20 then
!!    fCallback.WorkFinished(fWorkName,GetTickCount64-tix) else
!!    fCallback.WorkFailed(fWorkName,'expected random failure');
!end;
The callback is expected to be supplied as a {\f1\fs20 ILongWorkCallback} interface instance, then stored in a {\f1\fs20 fCallback} protected field for further notification.\line Some work is done in the {\f1\fs20 TLongWorkServiceThread.Execute} method (here just a {\f1\fs20 Sleep()} of more than 5 seconds), and the end-of-work notification is processed, as success or failure (depending on random in this fake process class), on either of the {\f1\fs20 ILongWorkCallback} interface methods.
The following {\f1\fs20 class} will define, implement and register the {\f1\fs20 ILongWorkService} service on the server side:
!type
!  TLongWorkService = class(TInterfacedObject,ILongWorkService)
!  protected
!    fTotalWorkCount: Integer;
!  public
!    procedure StartWork(const workName: string; const onFinish: ILongWorkCallback);
!    function TotalWorkCount: Integer;
!  end;
!
!procedure TLongWorkService.StartWork(const workName: string;
!  const onFinish: ILongWorkCallback);
!begin
!  InterlockedIncrement(fTotalWorkCount);
!!  TLongWorkServiceThread.Create(workName,onFinish);
!end;
!
!function TLongWorkService.TotalWorkCount: Integer;
!begin
!  result := fTotalWorkCount;
!end;
!
!var HttpServer: TSQLHttpServer;
!    Server: TSQLRestServerFullMemory;
!begin
!  Server := TSQLRestServerFullMemory.CreateWithOwnModel([]);
!!  Server.ServiceDefine(TLongWorkService,[ILongWorkService],sicShared);
!  HttpServer := TSQLHttpServer.Create('8888',[Server],'+',useBidirSocket);
!!  HttpServer.WebSocketsEnable(Server,PROJECT31_TRANSMISSION_KEY);
!...
Purpose of those methods is just to create and launch the {\f1\fs20 TLongWorkServiceThread} process from a client request, then maintain a total count of started works, in a {\f1\fs20 sicShared} service instance - see @92@ - hosted in a {\f1\fs20 useBidirSocket} kind of HTTP server.
We have to explicitly call {\f1\fs20 TSQLHttpServer.WebSocketsEnable()} so that this server will be able to upgrade to our {\i WebSockets} protocol, using our binary framing, and the very same symmetric @*encryption@ key as on the client side - shared as a {\f1\fs20 PROJECT31_TRANSMISSION_KEY} constant in the sample, but which may be safely stored on both sides.
:173  Publish-subscribe for events
In @*event-driven@ architectures, the {\i @**publish-subscribe@} messaging pattern is a way of letting senders (called {\i publishers}) transmit messages to their receivers (called {\i subscribers}), without any prior knowledge of who those subscribers are. In practice, the {\i subscribers} will express interest for a set of messages, which will be sent by the {\i publisher} to all the {\i subscribers} of a given message, as soon as it is be notified.
\graph PublishSubscribe1 Publish-Subscribe Pattern
\Publisher\Subscriber 1\Event
\Publisher\Subscriber 2\Event
\Publisher\Subscriber 3
\
In our @63@ implementation, messages are gathered in {\f1\fs20 interface} types, and each message defined as a single method, their content being the methods parameters.\line Most of the SOA alternative (in Java or C#) do require class definition for messages. Our KISS approach will just use method parameters values as message definition.
To maintain a list of {\i subscribers}, the easiest is to store a {\i dynamic array} of {\f1\fs20 interface} instances, on the {\i publisher} side.
:   Defining the interfaces
We will now implement a simple {\i chat} service, able to let several clients communicate together, broadcasting any message to all the other connected instances. This sample is also located in the the {\f1\fs20 SQLite3\\Samples\\31 - WebSockets} sub-folder, as {\f1\fs20 Project31ChatServer.dpr} and {\f1\fs20 Project31ChatClient.dpr}.
So you first define the callback interface, and the service interface:
!type
!  IChatCallback = interface(IInvokable)
!    ['{EA7EFE51-3EBA-4047-A356-253374518D1D}']
!    procedure NotifyBlaBla(const pseudo, msg: string);
!  end;
!
!  IChatService = interface(IInvokable)
!    ['{C92DCBEA-C680-40BD-8D9C-3E6F2ED9C9CF}']
!    procedure Join(const pseudo: string; const callback: IChatCallback);
!    procedure BlaBla(const pseudo,msg: string);
!    procedure CallbackReleased(const callback: IInvokable; const interfaceName: RawUTF8);
!  end;
The main command of the {\f1\fs20 IChatService} service is {\f1\fs20 BlaBla()}, which should be propagated to all client instance having {\f1\fs20 Join}ed the conversation, via {\f1\fs20 IChatCallback.NotifyBlabla()} events.
Those interface types will be shared by both server and client sides, in the common {\f1\fs20 Project31ChatCallbackInterface.pas} unit. The definition is pretty close to what we wrote when @152@. For instance, if 3 people did join the chat room, the following process should take place:
\graph PublishSubscribeChat Chat Application using Publish-Subscribe
\Any Joined Client\IChatService\BlaBla()
\IChatService\IChatCallback 1\NotifyBlaBla()
\IChatService\IChatCallback 2\NotifyBlaBla()
\IChatService\IChatCallback 3
\
The only additional method is {\f1\fs20 IChatServer.CallbackReleased()}, which, by convention, will be called on the server side when any {\f1\fs20 callback} interface instance is released on the client side.
As such, the {\f1\fs20 IChatService.Join()} method will implement the {\i subscription} to the chat service, whereas {\f1\fs20 IChatServer.CallbackReleased()} will be called when the client-side callback instance will be released (i.e. when its variable will be assigned to {\f1\fs20 nil}), to {\i unsubscribe} for the chat service.
:   Writing the Publisher
On the server side, each call to {\f1\fs20 IChatService.Join()} will {\i subscribe} to an internal list of connections, simply stored as an {\f1\fs20 array of IChatCallback}:
!type
!  TChatService = class(TInterfacedObject,IChatService)
!  protected
!!    fConnected: array of IChatCallback;
!  public
!    procedure Join(const pseudo: string; const callback: IChatCallback);
!    procedure BlaBla(const pseudo,msg: string);
!    procedure CallbackReleased(const callback: IInvokable; const interfaceName: RawUTF8);
!  end;
!
!procedure TChatService.Join(const pseudo: string;
!  const callback: IChatCallback);
!begin
!!  InterfaceArrayAdd(fConnected,callback);
!end;
The {\f1\fs20 InterfaceArrayAdd()} function, as defined in {\f1\fs20 SynCommons.pas}, is a simple wrapper around any {\i dynamic array} of {\f1\fs20 interface} instances, so that you may use it, or the associated {\f1\fs20 InterfaceArrayFind()} or {\f1\fs20 InterfaceArrayDelete()} functions, to maintain the list of subscriptions.
Then a remote call to the {\f1\fs20 IChatService.BlaBla()} method should be broadcasted to all connected clients, just by calling the {\f1\fs20 IChatCallback.BlaBla()} method:
!procedure TChatService.BlaBla(const pseudo,msg: string);
!var i: integer;
!begin
!  for i := 0 to high(fConnected) do
!!    fConnected[i].NotifyBlaBla(pseudo,msg);
!end;
Note that every call to {\f1\fs20 IChatCallback.BlaBla()} within the loop will be made via {\i @*WebSockets@}, in an asynchronous and non blocking way, so that even in case of huge number of clients, the {\f1\fs20 IChatService.BlaBla()} method won't block. In case of high numbers of messages, the framework is even able to {\i gather} push notification messages into a single bigger message, to reduce the resource use - see @153@.
If you are a bit paranoid, you may ensure that the notification process will continue, if any of the event failed:
!procedure TChatService.BlaBla(const pseudo,msg: string);
!var i: integer;
!begin
!!  for i := high(fConnected) downto 0 do // downwards for InterfaceArrayDelete()
!    try
!      fConnected[i].NotifyBlaBla(pseudo,msg);
!!    except
!!      InterfaceArrayDelete(fConnected,i); // unsubscribe the callback on failure
!    end;
!end;
This safer implementation will unregister any failing callback. If the notification raised an exception, it will ensure that this particular invalid subscriber won't be notified any more. Note that since we may reduce the {\f1\fs20 fConnected[]} array size on the fly, the loop is processed {\i downwards}, to avoid any access violation.
On the server side, the service implementation has been registered as such:
!  Server.ServiceDefine(TChatService,[IChatService],sicShared).
!    SetOptions([],[optExecLockedPerInterface]);
Here, the {\f1\fs20 optExecLockedPerInterface} option has been set, so that all method calls will be made thread-safe: concurrent access to the internal {\f1\fs20 fConnected[]} list will be protected by a lock. Since a global list of connections is to be maintained, the service life time has been defined as {\f1\fs20 sicShared} - see @92@.
The following method will be called by the server, when a client callback instance is released (either explicitly, or if the connection is broken), so could be used to {\i unsubscribe} to the notification, simply by deleting the callback from the internal {\f1\fs20 fConnected[]} array:
!procedure TChatService.CallbackReleased(const callback: IInvokable; const interfaceName: RawUTF8);
!begin
!!  if interfaceName='IChatCallback' then
!!    InterfaceArrayDelete(fConnected,callback);
!end;
The framework will in fact recognize the following method definition in any {\f1\fs20 interface} type for a service (it will check the method name, and the method parameters):
!   procedure CallbackReleased(const callback: IInvokable; const interfaceName: RawUTF8);
When a callback {\f1\fs20 interface} parameter (in our case, {\f1\fs20 IChatCallback}) will be released on the client side, this method will be called with the corresponding {\f1\fs20 interface} instance and type name as parameters. You do not have to call explicitly any method on the client side to {\i unsubscribe} a service: assigning {\i nil} to a callback variable, or freeing the {\f1\fs20 class} instance owning it as a field on the {\i subscriber} side, will automatically unregister it on the {\i publisher} side.
:   Consuming the service from the Subscriber side
On the client side, you implement the {\f1\fs20 IChatCallback} callback interface:
!type
!  TChatCallback = class(TInterfacedCallback,IChatCallback)
!  protected
!    procedure NotifyBlaBla(const pseudo, msg: string);
!  end;
!
!procedure TChatCallback.NotifyBlaBla(const pseudo, msg: string);
!begin
!  writeln(#13'@',pseudo,' ',msg);
!end;
The {\f1\fs20 TInterfacedCallback} type defines a {\f1\fs20 TInterfacedObject} sub-class, which will automatically notify the REST server when it is released. By providing the client {\f1\fs20 TSQLRest} instance to the {\f1\fs20 TChatCallback.Create()} constructor, you will ensure that the {\f1\fs20 IChatService.CallbackReleased} method will be executed on the server side, when the {\f1\fs20 TChatCallback}/{\f1\fs20 IChatCallback} instance will be released on the client side.
Then you subscribe to your remote service as such:
!!var Service: IChatService;
!!    callback: IChatCallback;
!...
!    Client.ServiceDefine([IChatService],sicShared);
!!    if not Client.Services.Resolve(IChatService,Service) then
!      raise EServiceException.Create('Service IChatService unavailable');
!...
!!      callback := TChatCallback.Create(Client,IChatCallback);
!!      Service.Join(pseudo,callback);
!...
!    try
!      repeat
!        readln(msg);
!        if msg='' then
!          break;
!!        Service.BlaBla(pseudo,msg);
!      until false;
!    finally
!!      callback := nil; // will unsubscribe from the remote publisher
!      Service := nil;  // release the service local instance BEFORE Client.Free
!    end;
You could easily implement more complex {\i publish/subscribe} mechanisms, including filtering, time to live or tuned broadcasting, by storing some additional information to the {\f1\fs20 interface} instance (e.g. some value to filter, a timestamp). A dynamic array of dedicated {\f1\fs20 record}s - see @48@, or a list of {\f1\fs20 class} instances, may be used to store the {\i subscribers} expectations.
:   Subscriber multiple redirection
Sometimes, in a complex business system, you will define several uncoupled parts of your code subscribing to the same service events. In a DDD architecture, it will be typically happen when several domain bounded contexts subscribe to a single event source, implemented in the infrastructure layer.
The easiest implementation path is to have each part registering from its side. But it will induce some redundant traffic with the publisher. And it will most probably end-up with duplicated code on subscribers side.
You may try {\f1\fs20 TSQLRest.MultiRedirect} and register once to a remote service, then use an internal registration mechanism to have every part of your business logic registering and consuming the events. The method returns an {\f1\fs20 IMultiCallbackRedirect} interface, allowing registration of sub-callbacks, with an optional set of method names, if only a sub-set of events are needed.
Note that sub-callbacks do not need to inherit from the {\f1\fs20 TInterfacedCallback} type: a regular {\f1\fs20 TInterfacedObject} is enough. They will be automatically unregistered from the internal list, if they raise an exception.
:196   Proper threaded implementation
A {\i mORMot} @*multi-thread@ed server will use critical sections to protect shared data, and avoid potential race conditions. But even on client side, callbacks will be executed in the context of the {\i @*WebSockets@} transmission thread. And in a typical micro-services or event-driven architecture, most nodes are clients and servers at the same time, creating a peer-to-peer mesh of services. So you should prevent any race conditions in each and every node, by protecting access to any shared data.
Likewise, if your callback triggers another method which shares the same critical section in another thread, you may encounter @**deadlock@ issues. If an event triggers a callback within a critical section used to protect a shared resource, and if this callback runs a blocking REST request, the REST answer will be received in the context of the transmission thread. If this answer tries to access the same shared resource, there will be a conflict with the main critical section lock, so the execution will lock.
To implement proper thread-safety of your callback process you could follow some patterns.
- Use several small critical sections, protecting any shared data, with the smallest granularity possible. You may use {\f1\fs20 TSynLocker} mutex or {\f1\fs20 TLockedDocVariant} schema-less storage.
- In your regression tests, ensure you run multi-threaded scenarios, with parallel requests. You may find in {\f1\fs20 TSynParallelProcess} an easy way of running concurrent client/server tests. It will help finding out most obvious implementation issues.
- By definition, most deadlocks are difficult to reproduce - they are some kind of "Heisenbugs". You may ensure proper logging of the callback process, so that you will be able to track for any deadlock which may occur on production.
- A good idea may be to gather all non-blocking callback process in a background thread using {\f1\fs20 TSQLRest.AsynchRedirect}. This method will implement any interface via a fake class, which will redirect all methods calls into calls of another interface, but as a FIFO in a background thread. So you will ensure that all callback process will take place in a single thread, avoiding most concurrency issues. As a side effect, the internal FIFO will leverage other threads, so may help scaling your system. For a client application using some User Interface, see @191@ a lock-free alternative.
Multi-threading is the key to performance. But it is also hard to properly implement. By following those simple rules, you may reduce the risk of concurrency issues.
:191   Interacting with UI/VCL
As we have stated, all callback notifications do take place in the transmission thread, i.e. in the {\f1\fs20 TWebSocketProcessClientThread} instance corresponding to each connected client.
You may be tempted to use the VCL {\f1\fs20 @*Synchronize@()} method, as usual, to forward the notifications to the UI layer. Unfortunately, this may trigger some unexpected concurrency issue, e.g. when asynchronous notifications (e.g. {\f1\fs20 TChatCallback.NotifyBlaBla()}) are received during a blocking REST command (e.g. {\f1\fs20 Service.BlaBla()}). The {\f1\fs20 Synchronize} call within the blocking command will avoid any incoming asynchronous notification wait for the main thread to be available, and will block the reception of the answer of the pending REST command...\line If you experiment random hangouts of your User Interface, and {\f1\fs20 404 errors} corresponding to a low-level {\i WebSockets} timeout, even when closing the application, you have certainly hit such a @*deadlock@.
Get rid of all your {\f1\fs20 Synchronize()} calls! Use {\i Windows} messages instead: they are safe, efficient and fast. The framework allows to forward all incoming notifications as a dedicated {\i Windows} message in a single line:
! Client.ServiceNotificationMethodViaMessages(MainForm.Handle,WM_SERVICENOTIFICATION);
The {\f1\fs20 WM_SERVICENOTIFICATION} should have been defined as a custom user message:
!const
! WM_SERVICENOTIFICATION = WM_USER; // may be WM_USER+1, +2, ...
Then, the {\f1\fs20 TFormMain} should execute the message, as a regular event handler:
! TFormMain = class(TForm)
!...
! procedure ServiceNotification(var Msg: TMessage); message WM_SERVICENOTIFICATION;
!...
!procedure TFormMain.ServiceNotification(var Msg: TMessage);
!begin
!  TSQLRestClientURI.ServiceNotificationMethodExecute(Msg);
!end;
Thanks to these two lines, the callbacks will be executed asynchronously in the main UI thread, using the optimized {\i Message} queue of the Operating System, without any blocking execution, nor race condition.
:  Interface callbacks instead of class messages
If you compare with existing client/server SOA solutions (in Delphi, Java, C# or even in Go or other frameworks), this {\f1\fs20 interface}-based callback mechanism sounds pretty unique and easy to work with.
Most {\i Events Oriented} solutions do use a set of dedicated messages to propagate the events, with a centralized {\i Message Bus} (like {\i MSMQ} or {\i JMS}), or a P2P approach (see e.g. {\i ZeroMQ} or {\i NanoMsg}). In practice, you are expected to define one {\f1\fs20 class} per message, the {\f1\fs20 class} fields being the message values. You will define e.g. one {\f1\fs20 class} to notify a successful process, and another {\f1\fs20 class} to notify an error. @*SOA@ services will eventually tend to be defined by a huge number of individual classes, with the temptation of re-using existing classes in several contexts.
Our {\f1\fs20 interface}-based approach allows to gather all messages:
- In a single {\f1\fs20 interface} type per {\i notification}, i.e. probably per {\i service operation};
- With one {\i method} per {\i event};
- Using method {\i parameters} defining the event {\i values}.
Since asynchronous notifications are needed most of the time, method parameters will be one-way, i.e. only {\f1\fs20 const}. Blocking request may also be defined, as we will see @181@. And an evolved algorithm will transparently gather outgoing messages, to enhance scalability.
Behind the scene, the framework will still transmit raw messages over IP sockets, like other systems, but events notification will benefit from using interfaces, on both server and client sides.
:   Using service and callback interfaces
For instance, you may define the following generic service and callback to retrieve a file from a remote camera, using {\i mORMot}'s {\f1\fs20 interface}-based approach:
!type
!  // define some custom types to make the implicit explicit
!  TCameraID = RawUTF8;
!  TPictureID = RawUTF8;
!  // mORMot notifications using a callback interface definition
!  IMyCameraCallback = interface(IInvokable)
!    ['{445F967F-79C0-4735-A972-0BED6CC63D1D}']
!    procedure Started(const Camera: TCameraID; const Picture: TPictureID);
!    procedure Progressed(const Camera: TCameraID; const Picture: TPictureID;
!      CurrentSize,TotalSize: cardinal);
!    procedure Finished(const Camera: TCameraID; const Picture: TPictureID;
!      const PublicURI: RawUTF8; TotalSize: cardinal);
!    procedure ErrorOccured(const Camera: TCameraID; const Picture: TPictureID;
!      const MessageText: RawUTF8);
!  end;
!  // mORMot main service, also defined as an interface
!  IMyCameraService = interface(IInvokable)
!    ['{3CE61E74-A01D-41F5-A414-94F204F140E1}']
!    function TakePicture(const Camera: TCameraID; const Callback: IMyCameraCallback): TPictureID;
!  end;
Take a deep breath, and keep in mind those two type definitions as reference. In a single look, I guess you did get the expectation of the "Camera Service". We will now compare with a classical message-based pattern.
:   Classical message(s) event
With a {\f1\fs20 class}-based message kind of implementation, you will probably define a single {\f1\fs20 class}, containing all potential information:
!type
!  // a single class message will need a status
!  TMyCameraCallbackState = (
!    ccsStarted, ccsProgressed, ccsFinished, ccsErrorOccured);
!  // the single class message
!  TMyCameraCallbackMessage = class
!  private
!    fCamera: TCameraID;
!    fPicture: TPictureID;
!    fTotalSize: cardinal;
!    fMessageText: RawUTF8;
!    fState: TMyCameraCallbackState;
!  published
!    property State: TMyCameraCallbackState read fState write fState;
!    property Camera: TCameraID read fCamera write fCamera;
!    property Picture: TPictureID read fPicture write fPicture;
!    property TotalSize: cardinal read fTotalSize write fTotalSize;
!    property MessageText: RawUTF8 read fMessageText write fMessageText;
!  end;
This single class is easy to write, but makes it a bit confusing to consume the notification. Which field comes with which state? The client-side code will eventually consist of a huge {\f1\fs20 case aMessage.State of} ... block, with potential issues. The business logic does not appear in this type definition. Easy to write, difficult to read - and maintain...
In order to have an implementation closer to @47@, you may define a set of classes, as such:
!type
!  // all classes will inherit from this one, to have common properties
!  TMyCameraCallbackAbstract = class
!  private
!    fCamera: TCameraID;
!    fPicture: TPictureID;
!  published
!    property Camera: TCameraID read fCamera write fCamera;
!    property Picture: TPictureID read fPicture write fPicture;
!  end;
!  // message class when the picture acquisition starts
!  TMyCameraCallbackStarted = class(TMyCameraCallbackAbstract);
!  // message class when the picture is acquired
!  TMyCameraCallbackFinished = class(TMyCameraCallbackAbstract)
!  private
!    fPublicURI: RawUTF8;
!    fTotalSize: cardinal;
!  published
!    property TotalSize: cardinal read fTotalSize write fTotalSize;
!    property PublicURI: RawUTF8 read fPublicURI write fPublicURI;
!  end;
!  // message during picture download
!  TMyCameraCallbackProgressed = class(TMyCameraCallbackFinished)
!  private
!    fCurrentSize: cardinal;
!  published
!    property CurrentSize: cardinal read fCurrentSize write fCurrentSize;
!  end;
!  // error message
!  TMyCameraCallbackErrorOccured = class(TMyCameraCallbackAbstract)
!  private
!    fMessageText: RawUTF8;
!  published
!    property MessageText: RawUTF8 read fMessageText write fMessageText;
!  end;
Inheritance makes this class hierarchy not as verbose as it may have been with plain "flat" classes, but it is still much less readable than the {\f1\fs20 IMyCameraCallback} type definition.
In both cases, such {\f1\fs20 class} definitions make it difficult to guess to which message matches which service. You must be very careful and consistent about your naming conventions, and uncouple your service definitions in clear name spaces.
When implementing @*SOA@ services, @*DDD@'s {\i @*Ubiquitous Language@} tends to be polluted by the {\f1\fs20 class} definition (getters and setters), and implementation details of the messages-based notification: your {\i Domain} code will be tied to the message oriented nature of the {\i Infrastructure} layer. We will see @172@ how {\f1\fs20 interface} callbacks will help implementing DDD's {\i @*Event-Driven@} pattern, in a cleaner way.
:181   Workflow adaptation
Sometimes, it may be necessary to react to some unexpected event. The consumer may need to change the workflow of the producer, depending on some business rules, an unexpected error, or end-user interaction.
By design, message-based implementations are asynchronous, and non-blocking: messages are sent and stored in a message broker/bus, and its internal processing loop propages the messages to all subscribers. In such an implementation, there is no natural place for "reverse" feedback messages.
A common pattern is to have a dedicated set of "answer/feedback" messages, to notify the service providers of a state change - it comes with potential race conditions, or unexpected rebound phenomenons, for instance when you add a node to an existing event-driven system.
Another solution may be to define explicit {\i rules} for service providers, e.g. when the service is called. You may define a set of workflows, injected to the provider/bus service at runtime. It will definitively tend to break the @182@, and put logic in the infrastructure layer.
On the other hand, since {\i mORMot}'s callbacks are true {\f1\fs20 interface} methods, they may return some values (as a {\f1\fs20 function} result or a {\f1\fs20 var/out} parameter). On the server side, such callbacks will block and wait for the client end to respond.
So by writing an additional method like:
!  IMyCameraCallback = interface(IInvokable)
!  ...
!    function ShouldRetryIfBusy(const Camera: TCameraID; const Picture: TPictureID): boolean;
!  ...
... you will be able to implement any needed complex workflow adaptation, in real time. The server side code will still be very readable and efficient, with no complex plumbing, wait queue or state machine to set up.
:   From interfaces comes abstraction and ease
As an additional benefit, integration with the Delphi language is clearly implementation agnostic: you are not even tied to use the framework, when working with such {\f1\fs20 interface} type definitions. In fact, this is a good way of implementing callbacks conforming to @47@ on the server side, and let the {\i mORMot} framework publish this mechanism in a client/server way, by using {\i WebSockets}, only if necessary.
The very same code could be used on the server side, with no transmission nor marshalling overhead (via direct {\f1\fs20 interface} instance calls), and over a network, with optimized use of resource and bandwidth (via "fake" {\f1\fs20 interface} calls, and binary/JSON marshalling over TCP/IP).
On the server side, your code - especially your {\i Domain} code - may interact directly with the lower level services, defined in the {\i Domain} as {\f1\fs20 interface} types, and implemented in the {\i infrastructure} layer. You may host both {\i Domain} and {\i Infrastructure} code in a single server executable, with direct assignment of local {\f1\fs20 class} instance as callbacks. This will minimize the program resources, in both CPU and memory terms - which is always a very valuable goal, for any business system.
You may be able to reuse your application and business logic in a stand-alone application, with similar direct calls from the UI to the application {\f1\fs20 interface}. On need, the {\f1\fs20 interface} variable may point to a remote {\i mORMot} server, without touching VCL/FMX code.
Last but not least, using an {\f1\fs20 interface} will help implementing the whole callback mechanism using @166@, e.g. for easy unit testing via @180@.\line You may also write your unit tests with real local callback {\f1\fs20 class} instances, which will be much easier to debug than over the whole client/server stack. Once you identified a scenario which fails the system, you could reproduce it with a dedicated test, even in an aggressive multi-threaded way, then use the debugger to trace the execution and identify the root cause of the issue.
\page
: Implementation details
:165  Error handling
Usually, in Delphi applications (like in most high-level languages), errors are handled via {\i exceptions}. By default, any {\f1\fs20 Exception} raised on the server side, within an {\f1\fs20 interface}-based service method, will be intercepted, and transmitted as an HTTP error to the client side, then a safe but somewhat obfuscated {\f1\fs20 EInterfaceFactoryException} will be raised, containing additional information serialized as JSON.
You may wonder why exceptions are not transmitted and raised directly on the client side, as if they were executed locally.\line In fact, {\f1\fs20 Exceptions} are not {\i value} objects, but true {\f1\fs20 class} instances, with some methods and potentially internal references to other objects. Most of the time, they are tied to a particular execution context, and even some low-level implementation details. A Delphi {\f1\fs20 exception} is even something very specific, and will not be easily converted into e.g. a {\i JavaScript}, {\i Java} or C# exception.
In practice, re-creating and raising an instance of the same {\f1\fs20 Exception class} which occurred on the server side will induce a strong dependency of the client code towards the server implementation details. For instance, if the server side raises a {\f1\fs20 ESQLDBOracle} exception, translating it on the other end will link your client side with the whole {\f1\fs20 SynDBOracle.pas} unit, which certainly not worth it. The {\f1\fs20 ESQLDBOracle} exception, by itself, contains a link to an {\i Oracle} statement instance, which will be lost when transmitted over the wire. Some client platforms (e.g. mobile or AJAX) do not even have any knowledge of what an {\i Oracle} database is...\line As such, {\f1\fs20 exception} are not good candidate on serialization, and transmission per value, from the server side to the client side. We will NOT be in favor of propagating exceptions to the client side.
This is why exceptions should better be intercepted on the server side, with a {\f1\fs20 try .. except} block within the service methods, then converted into low level DTO types, specific to the service, then explicitly transmitted as error codes to the client.
The first rule is that raising {\f1\fs20 exception} should be {\i exceptional} - as its name states: {\f1\fs20 exception}al. I mean, service code should not raise an {\f1\fs20 exception} in normal execution, even in case of wrong input. For instance, a wrong input parameter should lead into an application level error, transmitted as an enumeration item and/or some additional (probably text) information, but the business logic should never raise any {\f1\fs20 exception}. Only in case of low-level unexpected event (e.g. a SQL level failure, a GPF or Access Violation, a communication error with another trusted internal service), the server side may enter in {\i panic} mode, and raise an {\f1\fs20 exception}. Remember that {\f1\fs20 exception}s are intercepted by {\f1\fs20 SynLog.pas} and can be easily logged by our @16@: you will be able to identify the execution context, and find a full stack trace of the issue. But most common errors should be handled at business logic level, even defined in each service layers.
In practice, you may use an {\i enumerate}, in conjunction with a {\f1\fs20 variant} for additional structured information (as a {\f1\fs20 string} or a more complex {\f1\fs20 TDocVariant}), to transmit an error to the client side. You may define dedicated types at every layer, e.g. with {\f1\fs20 interface} types for {\i Domain} services, or {\i Application} services.
See for instance how {\f1\fs20 ICQRSService}, and its associated {\f1\fs20 TCQRSResult} enumeration, are defined in {\f1\fs20 mORMotDDD.pas}:
!type
!  TCQRSResult =
!    (cqrsSuccess, cqrsSuccessWithMoreData,
!     cqrsUnspecifiedError, cqrsBadRequest,
!     cqrsNotFound, cqrsNoMoreData, cqrsDataLayerError,
!     cqrsInternalError, cqrsDDDValidationFailed,
!     cqrsInvalidContent, cqrsAlreadyExists,
!     ...
!
!  ICQRSService = interface(IInvokable)
!    ['{923614C8-A639-45AD-A3A3-4548337923C9}']
!    function GetLastError: TCQRSResult;
!    function GetLastErrorInfo: variant;
!  end;
The first {\f1\fs20 cqrsSuccess} item of the {\f1\fs20 TCQRSResult} enumerate will be the default one (mapped and transmitted to a 0 JSON number), so in case of any stub or mock of the interfaces, fake methods will return as successful, as expected - see @166@.
When any {\f1\fs20 exception} is raised in a service method, a {\f1\fs20 TCQRSResult} enumeration value can be returned as result, so that error will be transmitted directly:
!function TDDDMonitoredDaemon.Stop(out Information: variant): TCQRSResult;
!...
!begin
!!  CqrsBeginMethod(qaNone,result);
!  try
!....
!    CqrsSetResult(cqrsSuccess,result);
!  except
!!    on E: Exception do
!!      CqrsSetResult(E,cqrsInternalError,result);
!  end;
!end;
But such {\f1\fs20 exception} should be {\f1\fs20 exception}al, as we already stated.
The {\f1\fs20 mORMotDDD.pas} unit defines, in the {\f1\fs20 TCQRSQueryObject} abstract {\f1\fs20 class}, some protected methods to handle errors and {\f1\fs20 exception}s as expected by {\f1\fs20 ICQRSService}. For instance, the {\f1\fs20 TCQRSQueryObject.CqrsSetResult()} method will set {\f1\fs20 result := cqrsInternalError} and serialize the {\f1\fs20 E: Exception} within the internal variant used for additional error, ready to be retrieved using {\f1\fs20 ICQRSService.GetLastErrorInfo}.
{\f1\fs20 Exception}s are very useful to interrupt a process in case of a catastrophic failure, but they are not the best method for transmitting errors over remote services. Some newer languages (e.g. {\i Google}'s {\f1\fs20 Go}), will even not define any {\f1\fs20 exception} type at language or RTL level, but rely on returned values, to transmit the errors in between execution contexts - see @https://golang.org/doc/faq#exceptions: in our client-server error handling design, we followed the same idea.
:77  Security
As stated in the features grid of @63@, a complete @*security@ pattern is available when using client-server services. In a @17@, securing messages between clients and services is essential to protecting data.
Security is implemented at several levels, following the main security patterns of {\i mORMot} - see @43@:
- {\i Process safety}, mainly for communication stream - e.g. when using HTTPS protocol at the @35@, or a custom cypher within HTTP content-encoding;
- At RESTful / URI {\i @*authentication@} level - see @18@ about {\i Session}, {\i Group} and {\i User} notions;
- Via {\i @*authorization@} at {\f1\fs20 interface} or method (service/operation) level to allow or forbid a given operation.
Let us discuss the two last points now ({\i authentication} and {\i authorization}).
By default, the settings are the following for interface-based services:
- All services (i.e. all interfaces) expect one {\i authentication scheme} to be validated (at least {\f1\fs20 TSQLRestServerAuthenticationWeak}), i.e. a light session to have been initiated by the client - in short, explicit authentication is mandatory;
- All operations (i.e. all methods) are allowed to execution - in short, authorization is enabled but opened.
You can chance these settings on the server side (it's an implementation detail - so it does not make any sense to tune it on the client side) via the {\f1\fs20 TServiceFactoryServer} instance corresponding to each {\f1\fs20 interface}. You can access those instances e.g. from the {\f1\fs20 TSQLRestServer.Services} property.
To disable the whole service / {\f1\fs20 interface} need of authentication, you can use the {\f1\fs20 ByPassAuthentication} property of the {\f1\fs20 TServiceFactoryServer} instance corresponding to a given {\f1\fs20 interface}. It may be useful e.g. for simple web services which do not expose any sensitive data (e.g. a service catalog, or a service returning public information or even HTML content).
Then, to tune the authorization process at operational (method) level, {\f1\fs20 TServiceFactoryServer} provides the following methods to change the security policy for each {\f1\fs20 interface}:
- {\f1\fs20 AllowAll()} and {\f1\fs20 Allow()} to enable methods execution globally;
- {\f1\fs20 DenyAll()} and {\f1\fs20 Deny()} to disable methods execution globally;
- {\f1\fs20 AllowAllByID()} and {\f1\fs20 AllowByID()} to enable methods execution by Group IDs;
- {\f1\fs20 DenyAllByID()} and {\f1\fs20 DenyByID()} to disable methods execution by Group IDs;
- {\f1\fs20 AllowAllByName()} and {\f1\fs20 AllowByName()} to enable methods execution by Group names;
- {\f1\fs20 DenyAllByName()} and {\f1\fs20 DenyByName()} to disable methods execution by Group names.
The first four methods will affect everybody. The next {\f1\fs20 *ByID()} four methods accept a list of {\i authentication Group} IDs (i.e. {\f1\fs20 TSQLAuthGroup.ID} values), where as the {\f1\fs20 *ByName()} methods will handle {\f1\fs20 TSQLAuthGroup.Ident} property values.
In fact, the execution can be authorized for a particular group of authenticated users. Your service can therefore provide some basic features, and then enables advanced features for administrators or supervisors only. Since the User / Group policy is fully customizable in our RESTful authentication scheme - see @18@, {\i mORMot} provides a versatile and inter-operable security pattern.
Here is some extract of the supplied regression tests:
! (...)
!!  S := fClient.Server.Services['Calculator'] as TServiceFactoryServer;
!  Test([1,2,3,4,5],'by default, all methods are allowed');
!!  S.AllowAll;
!  Test([1,2,3,4,5],'AllowAll should change nothing');
!  S.DenyAll;
!  Test([],'DenyAll will reset all settings');
!  S.AllowAll;
!  Test([1,2,3,4,5],'back to full acccess for everybody');
!  S.DenyAllByID([GroupID]);
!  Test([],'our current user shall be denied');
!  S.AllowAll;
!  Test([1,2,3,4,5],'restore allowed for everybody');
!  S.DenyAllByID([GroupID+1]);
!  Test([1,2,3,4,5],'this group ID won''t affect the current user');
!  S.DenyByID(['Add'],GroupID);
!  Test([2,3,4,5],'exclude a specific method for the current user');
!  S.DenyByID(['totext'],GroupID);
!  Test([2,3,5],'exclude another method for the current user');
! (...)
In the above regression tests code, the {\f1\fs20 Test()} local procedure is used to validate the corresponding methods of {\f1\fs20 ICalculator} according to a set of method indexes (1={\i Add}, 2={\i Multiply}, 3={\i Subtract}, 4={\i ToText}...).
In this code, the {\f1\fs20 GroupID} value was retrieved as such:
!  GroupID := fClient.MainFieldID(TSQLAuthGroup,'User');
And the current authenticated user on the client side has been defined to be a member of the {\f1\fs20 'User'} group:
!  fClient.SetUser('User','synopse'); // default user for Security tests
Since {\f1\fs20 TSQLRestServer.ServiceRegister} and {\f1\fs20 TSQLRestServer.ServiceDefine} methods return the first created {\f1\fs20 TServiceFactoryServer} instance, and since all {\f1\fs20 Allow* / AllowAll* / Deny* / DenyAll*} methods return also a {\f1\fs20 TServiceFactoryServer} instance, you can use some kind of "fluent interface" in your code to set the security policy, as such:
!  Server.ServiceDefine(TServiceCalculator,[ICalculator],sicShared).
!    DenyAll.AllowAllByName(['Supervisor']);
This will allow access to the {\f1\fs20 ICalculator} methods only for the {\i Supervisor} group of users.
:  Implementation class types
Most of the time, your implementation class will inherit from {\f1\fs20 TInterfacedObject}. As stated above, you could in fact inherit from any plain {\i Delphi} class: the only condition is that it implements the expected interface, and has a @*GUID@.
But if you need a special process to take place during the class instance initialization, you can inherit from the {\f1\fs20 TInterfacedObjectWithCustomCreate} class, which provides the following virtual {\f1\fs20 constructor}, ready to be overridden with your customized initialization:
!  TInterfacedObjectWithCustomCreate = class(TInterfacedObject)
!  public
!    /// this virtual constructor will be called at instance creation
!    constructor Create; virtual;
!  end;
But from the @*SOA@ point of view, it could make sense to use a dedicated method with proper parameters to initialize your instance, e.g. in you are in {\f1\fs20 sicClientDriven} execution mode. See in @74@ some sample code implementing a {\f1\fs20 IRemoteSQL} service, with a dedicated {\f1\fs20 Connect()} method to be called before all other methods to initialize a {\f1\fs20 sicClientDriven} instance.
:72  Server-side execution options (threading)
When a service is registered on the server side, some options can be defined in order to specify its execution details, using the {\f1\fs20 TServiceFactoryServer.SetOptions()} method.
By default, service methods are called within the thread which received them. That is, when hosted by @*multi-thread@ed server instances (e.g. {\f1\fs20 TSQLite3HttpServer} or {\f1\fs20 TSQLRestServerNamedPipeResponse}), the method context can be re-entrant - unless it has been defined with {\f1\fs20 sicSingle} or {\f1\fs20 sicPerThread} instance lifetime modes. It allows better response time and CPU use, but drawback is that the method implementation shall be thread-safe. This is the technical reason why service implementation methods have to handle multi-threading safety carefully, e.g. by using @184@ on purpose.
The following execution options are available:
|%35%65
|\b TServiceMethodOptions|Description\b0
|{\i none} (default)|All methods are re-entrant and shall be coded to be thread-safe
|{\f1\fs20 optExecLockedPerInterface}|Each interface will be protected/locked by its own {\i mutex}
|{\f1\fs20 optExecInMainThread\line optFreeInMainThread}|Methods will be executed in the process main thread\line Interface will be released in the process main thread
|{\f1\fs20 optExecInPerInterfaceThread\line optFreeInPerInterfaceThread}|Each interface will execute its methods in its own thread\line Each interface will be freed in its own thread
|%
Of course, {\f1\fs20 SetOption()} accepts an optional list of method names, if you want to tune the execution at the method level.
Setting {\f1\fs20 optExecLockedPerInterface} option will {\i lock} the specified method(s) execution at the interface level. That is, it won't be possible to have two methods of the same interface be executed concurrently. This option uses a {\f1\fs20 TRTLCriticalSection} mutex, so is at the same time safe and using very little resources. But it won't guaranty that the method execution will always take place in the same thread: so if you need some per-thread initialization/finalization (e.g. for COM objects), you should better use the other options.
Setting {\f1\fs20 optExecInMainThread} option will force the specified method(s) to be called within a {\f1\fs20 RunningThread.Synchronize()} call - it can be used e.g. if your implementation rely heavily on COM objects, or if you want to ensure that your code will work correctly, without the need to worry about thread safety, which can be quite difficult to deal with. The {\f1\fs20 optFreeInMainThread} option will also ensure that the service class instance will be released in the main thread (i.e. its {\f1\fs20 Free} method called via {\f1\fs20 Synchronize}). Since the main thread will be used by all interfaces, it could result into an execution bottleneck.
Setting {\f1\fs20 optExecInPerInterfaceThread} option will force the specified method(s) to be called within a thread (to be more precise, a {\f1\fs20 TSynBackgroundThreadSQLRestServerProcedure} class, which will notify the {\f1\fs20 TSQLSQLRestServer} for the thread context) dedicated to the interface. An associated {\f1\fs20 optFreeInPerInterfaceThread} option will also ensure that the service class instance will be released in the same thread: it is pretty convenient to use this threading model, for instance if you want to maintain a dedicated {\f1\fs20 SynDB.pas}-based database connection, or initialize some COM objects.
For instance, if you want all the methods of your {\f1\fs20 TServiceCalculator} class to be executed in the main thread, you can define:
! Server.ServiceDefine(TServiceCalculator,[ICalculator],sicShared).
!  SetOptions([],[optExecInMainThread]);
Or if only the {\f1\fs20 TServiceCalculator.Add} method has to be protected, you can write:
! Server.ServiceDefine(TServiceCalculator,[ICalculator],sicShared).
!  SetOptions(['Add'],[optExecInMainThread]);
In fact, the {\f1\fs20 SetOptions()} method follows a call signature similar to the one used for defining the service security.
For best performance, you may define your service methods be called without any locking, but rely on some convenient classes defined in {\f1\fs20 SynCommons.pas} - as the {\f1\fs20 @*TAutoLocker@} class or the {\f1\fs20 @**TLockedDocVariant@} kind of storage, for efficient multi-thread process.\line A similar thread safety concern also applies to @*MVVM@ methods - see @111@.
:183  Audit Trail for Services
We have seen previously how the @*ORM@ part of the framework is able to provide an @85@. It is a very convenient way of storing the change of state of the data. On the other side, in any modern @*SOA@ solution, data is not at the center any more, but services. Sometimes, the data is not stored within your server, but in a third-party @17@. Being able to monitor the service execution of the whole system becomes sooner or later mandatory. Our framework allows to create an {\i @**Audit Trail@} of any incoming or outgoing service operation, in a secure, efficient and automated way.
:   When logging is not enough
By default, any {\f1\fs20 interface}-based service process will be logged by the framework - see @73@ - in dedicated {\f1\fs20 sllServiceCall} and {\f1\fs20 sllServiceReturn} log levels. You may see output similar to the following:
$18:03:18 Enter   mORMot.TSQLRestServerFullMemory(024500A0).URI(POST root/DomUserQuery.SelectByLogonName/1 inlen=7)
$!18:03:18 Service call      mORMot.TSQLRestServerFullMemory(024500A0) DomUserQuery.SelectByLogonName["979"]
$18:03:18 Server             mORMot.TSQLRestServerFullMemory(024500A0)   POST root/DomUserQuery.SelectByLogonName SOA-Interface -> 200 with outlen=21 in 16 us
$!18:03:18 Service return    mORMot.TSQLRestServerFullMemory(024500A0) {"result":[0],"id":1}
$18:03:18 Leave   00.000.017
The above lines match the execution of the following method, as defined in {\f1\fs20 dddDomUserCQRS.pas}:
!  IDomUserQuery = interface(ICQRSService)
!    ['{198C01D6-5189-4B74-AAF4-C322237D7D53}']
!    /// will select a single TUser from its logon name
!    // - then use Get() method to retrieve its content
!    function SelectByLogonName(const aLogonName: RawUTF8): TCQRSResult;
!  ...
The actual execution was:
! IDomUserQuery.SelectByLogonName('979') -> cqrsSuccess
Here {\f1\fs20 cqrsSuccess} is the first item of the enumeration result, returned as an integer JSON value {\f1\fs20 "result":[0]} by the method:
!  TCQRSResult =
!    (cqrsSuccess, cqrsSuccessWithMoreData,
!     cqrsUnspecifiedError, cqrsBadRequest, cqrsNotFound,
!    ...
This detailed log (including micro-second timing on the "{\i Leave}" rows) is very helpful for support, especially to investigate about any error occurring on a production server. But it will not be enough (or on the contrary provide "too much information" which "kills the information") to monitor the higher level of the process, especially on a server with a lot of concurrent activity.
:   Tracing Service Methods
The framework allows to optionally store each @*SOA@ method execution in a database, with the input and output parameters, and accurate timing.\line You could enable this automated process:
- Either at service level, using {\f1\fs20 TServiceFactoryServer.SetServiceLog()};
- Or for all services of a {\f1\fs20 TSQLRestServer.ServiceContainer} instance, via {\f1\fs20 TServiceContainerServer.SetServiceLog()}.
For instance, you may enable it for a whole @*REST@ server:
! (aRestSOAServer.ServiceContainer as TServiceContainerServer).SetServiceLog(
!   aRestLogServer,TSQLRecordServiceLog);
This single command will create an Audit Trail with all service calls made on {\f1\fs20 aRestSOAServer} to the {\f1\fs20 TSQLRecordServiceLog} ORM class of {\f1\fs20 aRestLogServer}. Keeping a dedicated REST server for the log entries will reduce the overhead on the main server, and ease its maintenance.
Actual storage takes place within a class inheriting from {\f1\fs20 TSQLRecordServiceLog}:
!  TSQLRecordServiceLog = class(TSQLRecord)
!  ...
!  published
!    /// the 'interface.method' identifier of this call
!    // - this column will be indexed, for fast SQL queries, with the MicroSec
!    // column (for performance tuning)
!    property Method: RawUTF8 read fMethod write fMethod;
!    /// the input parameters, as a JSON document
!    // - will be stored in JSON_OPTIONS_FAST_EXTENDED format, i.e. with
!    // shortened field names, for smaller TEXT storage
!    // - content may be searched using JsonGet/JsonHas SQL functions on a
!    // SQlite3 storage, or with direct document query under MongoDB/PostgreSQL
!    property Input: variant read fInput write fInput;
!    /// the output parameters, as a JSON document, including result: for a function
!    // - will be stored in JSON_OPTIONS_FAST_EXTENDED format, i.e. with
!    // shortened field names, for smaller TEXT storage
!    // - content may be searched using JsonGet/JsonHas SQL functions on a
!    // SQlite3 storage, or with direct document query under MongoDB/PostgreSQL
!    property Output: variant read fOutput write fOutput;
!    /// the Session ID, if there is any
!    property Session: integer read fSession write fSession;
!    /// the User ID, if there is an identified Session
!    property User: integer read fUser write fUser;
!    /// will be filled by the ORM when this record is written in the database
!    property Time: TModTime read fTime write fTime;
!    /// execution time of this method, in micro seconds
!    property MicroSec: integer read fMicroSec write fMicroSec;
!  end;
The @*ORM@ will therefore store the following table on its database:
\graph DBServiceLog ServiceLog Record Layout
rankdir=LR;
node [shape=Mrecord];
struct1 [label="ID : TID|Input : variant|Method : RawUTF8|MicroSec : integer|Output : variant|Session : integer|Time : TModTime|User : integer"];
\
As you can see, all input and output parameters are part of the record, as two {\f1\fs20 @*TDocVariant@} instances. Since they are stored as JSON/TEXT, you could perform some requests directly on their content, especially if actual storage take place in a {\i @*MongoDB@} database: you may even use dedicated indexes on the parameter values, and/or run advanced {\i @*map/reduce@} queries. You can use {\f1\fs20 optNoLogInput} or {\f1\fs20 optNoLogOutput} settings with {\f1\fs20 TInterfaceFactory.SetOptions()} to hide all input or output parameters values, or define some value types as containing {\i Sensitive Personal Information} (@*SPI@), using {\f1\fs20 TInterfaceFactory.RegisterUnsafeSPIType}.
Since very accurate timing, with a micro-second resolution, is part of the information, you will be able to make filtering or advanced statistics using simple SQL clauses. It has never been easier to monitor your @*SOA@ system, and identify potential issues. You may easily extract this information from your database, and feed a real-time visual monitoring chart system, for instance. Or identify and spy unusual execution patterns (e.g. unexpected timing or redounding error codes), which will match some SQL requests: those SQL statements may be run automatically on a regular basis, to prevent any problem before it actually happen.
:   Tracing Asynchronous External Calls
Sometimes, your server may be the client of another process. In an @*SOA@ environment, you may interface with a third-party @*REST@ service for an external process, e.g. sending a real-time notification.
On the REST client instance, you can execute the {\f1\fs20 TServiceFactoryClient.SendNotifications()} method for a given service:
!  aNotificationClientService.SendNotifications(aServicesLogRest,
!    TSQLRecordServiceNotifications, fSettings.NotificationsRetrySeconds);
This single command will create an Audit Trail with all notification calls sent to {\f1\fs20 aNotificationClientService}, in the {\f1\fs20 TSQLRecordServiceNotifications} ORM class of {\f1\fs20 aServicesLogRest}.
You may use the following {\f1\fs20 TSQLRecordServiceNotifications class}:
!  TSQLRecordServiceNotifications = class(TSQLRecordServiceLog)
!  ...
!  published
!    /// when this notification has been sent
!    // - equals 0 until it was actually notified
!    property Sent: TTimeLog read fSent write fSent;
!  end;
Which will be stored in the following table:
\graph DBServiceNotifications ServiceNotifications Record Layout
rankdir=LR;
node [shape=Mrecord];
struct1 [label="ID : TID|Sent : TTimeLog|Input : variant|Method : RawUTF8|MicroSec : integer|Output : variant|Session : integer|Time : TModTime|User : integer"];
\
The additional {\f1\fs20 Sent} property will contain the {\f1\fs20 TTimeLog} time-stamp on which the notification will have taken place.
In fact, all methods executed via this notification service will now be first stored in this table, then the remote HTTP notifications will take place asynchronously in the background. Transmission will be in order (first-in-first-out), and in case of any connection problem (e.g. the remote server not returning a {\f1\fs20 200 HTTP SUCCESS} status code), it won't move to the next entry, and will retry after the {\f1\fs20 NotificationsRetrySeconds} period, as supplied to the {\f1\fs20 SendNotifications()} method.
Of course, you may define your own sub-class, to customize the destination Audit Trail table:
!type
!  TSQLMyNotifications = class(TSQLRecordServiceNotifications);
Thanks to those {\f1\fs20 TSQLRecordServiceLog} classes, high-level support and analysis has never become easier. The actual implementation of those features has been tuned to minimize the impact on main performance, by using e.g. delayed write operations via @28@, or a dedicated background thread for the asynchronous notification process.
:  Transmission content
All data is transmitted as @*JSON@ arrays or objects, according to the requested URI.
We'll discuss how data is expected to be transmitted, at the application level.
:   Request format
As stated above, there are several available modes of routing, defined by a given {\f1\fs20 class}, inheriting from {\f1\fs20 TSQLRestServerURIContext}:
\graph HierTSQLRestServerURIContext Routing via TSQLRestServerURIContext classes hierarchy
\TSQLRestRoutingJSON_RPC\TSQLRestServerURIContext
\TSQLRestRoutingREST\TSQLRestServerURIContext
\
The corresponding description may be:
|%13%56%33
||\b {\f1\fs20 TSQLRestRoutingREST}|{\f1\fs20 TSQLRestRoutingJSON_RPC}\b0
|Mode|@*REST@ful|@*JSON-RPC@
|Default|Yes|No
|URI scheme|{\f1\fs20 /Model/Interface.Method[/ClientDrivenID]}\line or {\f1\fs20 /Model/Interface/Method[/ClientDrivenID]}\line + optional URI-encoded params|{\f1\fs20 /Model/Interface}
|Body content|JSON array of parameters\line or void if parameters were encoded at URI|{\f1\fs20 \{"method":"{\i MethodName}",\line  "params":[...]\line [,"id":{\i ClientDrivenID}]\}}
|Body content\line (alternative)|JSON object of parameters\line or void if parameters were encoded at URI|{\f1\fs20 \{"method":"{\i MethodName}",\line  "params":{...}\line [,"id":{\i ClientDrivenID}]\}}
|Security|RESTful @*authentication@ for each method\line or for the whole service (interface)|RESTful authentication for the whole service (interface)
|Speed|10% faster|10% slower
|%
Most of the time, the input parameters will be transmitted as a JSON array of values, following the exact order of {\f1\fs20 const} / {\f1\fs20 var} method parameters.\line As an alternative, a JSON object storing the input parameters by name will be accepted. This will be slightly slower than a JSON array of parameters, but could be handy, depending on the client side.\line Last but not least, {\f1\fs20 TSQLRestRoutingREST} is able to decode parameters encoded at URI level, as most regular historic HTTP requests.
The routing to be used is defined globally in the {\f1\fs20 TSQLRest.ServiceRouting} property, and should match on both client and server side, of course. By design, you should {\i never} assign the abstract {\f1\fs20 TSQLRestServerURIContext} to this property.
The {\f1\fs20 TSQLRestServerURIContext} abstract class defines the following methods, which will be overridden by inherited implementations to reflect the expected behavior on all aspects of the RESTful routing and transmission:
!  TSQLRestServerURIContext = class
!  protected
! ...
!    /// retrieve RESTful URI routing
!    function URIDecodeREST: boolean; virtual;
!    /// retrieve method-based SOA URI routing with optional RESTful mode
!    procedure URIDecodeSOAByMethod; virtual;
!    /// retrieve interface-based SOA
!    procedure URIDecodeSOAByInterface; virtual; abstract;
!    /// process authentication
!    function Authenticate: boolean; virtual;
!    /// direct launch of a method-based service
!    procedure ExecuteSOAByMethod; virtual;
!    /// direct launch of an interface-based service
!    procedure ExecuteSOAByInterface; virtual; abstract;
!    /// handle GET/LOCK/UNLOCK/STATE verbs for ORM/CRUD process
!    procedure ExecuteORMGet; virtual;
!    /// handle POST/PUT/DELETE/BEGIN/END/ABORT verbs for ORM/CRUD process
!    procedure ExecuteORMWrite; virtual;
! ...
Most of the time, the supplied {\f1\fs20 TSQLRestRoutingREST} and {\f1\fs20 TSQLRestRoutingJSON_RPC} classes will meet your requirements.
:    REST mode
:     Parameters transmitted as JSON array
In the default {\f1\fs20 TSQLRestRoutingREST} mode, both service and operation (i.e. interface and method) are identified within the URI. And the message body is a standard JSON array of the supplied parameters (i.e. all {\f1\fs20 const} and {\f1\fs20 var} parameters).
Here is typical request for {\f1\fs20 ICalculator.Add}:
$ POST /root/Calculator.Add
$ (...)
$ [1,2]
Here we use a {\f1\fs20 POST} verb, but the framework will also allows other methods like {\f1\fs20 GET}, if needed (e.g. from a regular browser). The pure {\i Delphi} client implementation will use only {\f1\fs20 POST}.
For a {\f1\fs20 sicClientDriven} mode service, the needed instance ID is appended to the URI:
$ POST /root/ComplexNumber.Add/1234
$ (...)
$ [20,30]
Here, {\f1\fs20 1234} is the identifier of the server-side instance ID, which is used to track the instance life-time, in {\f1\fs20 sicClientDriven} mode.  One benefit of transmitting the Client Session ID within the URI is that it will be more secure in our RESTful authentication scheme - see @18@: each method (and even any client driven session ID) will be signed properly.
:     Parameters transmitted as JSON object
The {\i mORMot} server will also accept the incoming parameters to be encoded as a JSON object of named values, instead of a JSON array:
$ POST /root/Calculator.Add
$ (...)
$ {"n1":1,"n2":2}
Of course, order of the values is not mandatory in a JSON object, since parameters will be lookup by name. As a result, the following request will be the same as the previous one:
$ POST /root/Calculator.Add
$ (...)
$ {"n2":2,"n1":1}
For a {\f1\fs20 sicClientDriven} mode service, the needed instance ID is appended to the URI:
$ POST /root/ComplexNumber.Add/1234
$ (...)
$ {"aReal":20,"aImaginary":30}
In some cases, naming the parameters could be useful, on the client side. But this should not be the default, since it will be slightly slower (for parsing and checking the names), and use more bandwidth at transmission.
Any missing parameter in the incoming JSON object will be replaced by its default value. For instance, the following will run {\f1\fs20 IComplexNumber.Add(0,2)}:
$ POST /root/Calculator.Add
$ (...)
$ {"n2":2}
Any unknown parameter in the incoming JSON object will just be ignored. It could be handy, if you want to transmit some generic execution context (e.g. a global "data scope" in a MVC model), and let the service use only the values it needs.
$ POST /root/ComplexNumber.Add/1234
$ (...)
$ {"Session":"1234","aImaginary":30,"aReal":20,"UserLogged":"Nikita"}
Of course, the extra values will consume some bandwidth for nothing, but the process cost on the server side will be negligible, since our implementation will just ignore those unexpected properties, without allocating any memory for them.
:     Parameters encoded at URI level
In this {\f1\fs20 TSQLRestRoutingREST} mode, the server is also able to retrieve the parameters from the URI, if the message body is left void. This is not used from a {\i Delphi} client (since it will be more complex and therefore slower), but it can be used for a client, if needed:
$ POST root/Calculator.Add?+%5B+1%2C2+%5D
$ GET root/Calculator.Add?+%5B+1%2C2+%5D
In the above line, {\f1\fs20 +%5B+1%2C2+%5D} will be decoded as {\f1\fs20 [1,2]} on the server side. In conjunction with the use of a {\f1\fs20 GET} verb, it may be more suitable for a remote @*AJAX@ connection.
As an alternative, you can encode and name the parameters at URI level, in a regular HTML fashion:
$ GET root/Calculator.Add?n1=1&n2=2
Since parameters are named, they can be in any order. And if any parameter is missing, it will be replaced by its default value (e.g. {\f1\fs20 0} for a number or {\f1\fs20 ''} for a {\f1\fs20 string}).
This may be pretty convenient for simple services, consumed from any kind of client.
Note that there is a known size limitation when passing some data with the URI over HTTP. Official RFC 2616 standard advices to limit the URI size to 255 characters, whereas in practice, it sounds safe to transmit up to 2048 characters within the URI. If you want to get rid of this limitation, just use the default transmission of a JSON array as request body.
As an alternative, the URI can be written as {\f1\fs20 /RootName/InterfaceName/MethodName}. It may be more RESTful-compliant, depending on your client policies. The following URIs will therefore be equivalent to the previous requests:
$ POST /root/Calculator/Add
$ POST /root/ComplexNumber/Add/1234
$ POST root/Calculator/Add?+%5B+1%2C2+%5D
$ GET root/Calculator/Add?+%5B+1%2C2+%5D
$ GET root/Calculator/Add?n1=1&n2=2
From a {\i Delphi} client, the {\f1\fs20 /RootName/InterfaceName.MethodName} scheme will always be used.
:     Sending a JSON object
By default, the {\i mORMot} client will send all values, transmitted as a JSON array without any parameter name, as we have seen:
$ POST /root/Calculator.Add
$ (...)
$ [1,2]
But if {\f1\fs20 TServiceFactoryClient.ParamsAsJSONObject} is set to {\f1\fs20 true}, the transmitted values from the client side will be encoded as a JSON object:
$ POST /root/Calculator.Add
$ (...)
$ {"n1":1,"n2":2}
This may help transmitting some values to a non-{\i mORMot} server, in another format, for a given service.
:197     Sending raw binary
If your purpose is to upload some binary data, {\f1\fs20 RawByteString} and {\f1\fs20 TSQLRawBlob} input parameters will by default be transmitted as @*Base64@ encoded JSON text.
You may define @49@ to transmit raw binary, without the Base64 encoding overhead. It would allow low-access to the input content type and encoding, even with multi-part file upload from HTTP.
As an alternative, if you use default {\f1\fs20 TSQLRestRoutingREST} routing, and defined a single {\f1\fs20 RawByteString} or {\f1\fs20 TSQLRawBlob} input parameter, it will be processed as a raw POST with binary body defined with mime-type {\f1\fs20 'application/octet-stream'}. This may be more optimized for remote access over the Internet.
:    JSON-RPC
:     Parameters transmitted as JSON array
If {\f1\fs20 TSQLRestRoutingJSON_RPC} mode is used, the URI will define the interface, and then the method name will be inlined with parameters, e.g.
$ POST /root/Calculator
$ (...)
$ {"method":"Add","params":[1,2],"id":0}
Here, the {\f1\fs20 "id"} field can be not set (and even not existing), since it has no purpose in {\f1\fs20 sicShared} mode.
For a {\f1\fs20 sicClientDriven} mode service:
$ POST /root/ComplexNumber
$ (...)
$ {"method":"Add","params":[20,30],"id":1234}
:     Parameters transmitted as JSON object
As an alternative, you may let the values be transmitted as a JSON object containing the named parameters values, instead of a JSON array:
$ POST /root/Calculator
$ (...)
$ {"method":"Add","params":{"n1":1,"n2":2},"id":0}
Here, the same rules applies than in {\f1\fs20 TSQLRestRoutingREST} mode:
- Any missing parameter will be replaced by its default value;
- Properties order is not sensitive anymore;
- Unexpected parameters will just be ignored.
Note that by definition, {\f1\fs20 TSQLRestRoutingJSON_RPC} mode is not able to handle URI-encoded parameters. In fact, the JSON-RPC mode expects the URI to be used only for identifying the service, and have the whole execution context transmitted as body.
:    REST mode or JSON-RPC mode?
For a standard {\i mORMot} Delphi client, or any supported {\i Cross-Platform} client - see @86@ - {\f1\fs20 TSQLRestRoutingREST} is preferred. The supplied libraries, even for {\i @*Smart Mobile Studio@}, fully implement this routing scheme. It is the faster, safer and most modular mode available.\line In practice, {\f1\fs20 TSQLRestRoutingJSON_RPC} mode has been found to be a little bit slower. Since the method name will be part of the URI, the signature will have a bigger extent than in JSON-RPC mode, so it will be more secure. Its ability to retrieve URI-encoded parameters could be also useful, e.g. to server some dynamic HTML pages in addition to the SOA endpoints, with proper HTTP caching abilities.
Of course, {\f1\fs20 TSQLRestRoutingJSON_RPC} mode may be used as an alternative, depending on the client expectations, and technology limitations, e.g. if your client expect a JSON-RPC compatible communication.\line It's up to you to select the right routing scheme to be used, depending on your needs.
:   Response format
:    Standard answer as JSON object
:     JSON answers
:      Returning as JSON array
The framework will always return the data in the same format, whatever the routing mode used.
Basically, this is a JSON object, with one nested {\f1\fs20 "result":} property, and the client driven {\f1\fs20 "id":} value (e.g. always 0 in {\f1\fs20 sicShared} mode):
$ POST /root/Calculator.Add
$ (...)
$ [1,2]
will be answered as such:
$ {"result":[3]}
For a {\f1\fs20 sicClientDriven, sicPerSession, sicPerUser, sicPerGroup} or {\f1\fs20 sicPerThread} mode service, the answer will contain an additional {\f1\fs20 "id":...} member, which will identify the corresponding session:
$ {"result":[3],"id":1234}
In {\f1\fs20 sicSingle} and {\f1\fs20 sicShared} modes, the {\f1\fs20 "id":0} member is just not emitted.
The {\i result} JSON array contains all {\f1\fs20 var} and {\f1\fs20 out} parameters values (in their declaration order), and then the method main result.
For instance, here is a transmission stream for a {\f1\fs20 ICalculator.ComplexCall} request in {\f1\fs20 TSQLRestRoutingREST} mode:
$ POST root/Calculator.ComplexCall
$ (...)
$ [[288722014,1231886296], ["one","two","three"], ["ABC","DEF","GHIJK"], "ïƒ¿BgAAAAAAAAAAAAAAAAAAACNEOlxEZXZcbGliXFNRTGl0ZTNcZXhlXFRlc3RTUUwzLmV4ZQ==", "ïƒ¿Xow1EdkXbUkDYWJj"]
will be answered as such:
$ '{"result":[ ["ABC","DEF","GHIJK","one,two,three"], "ïƒ¿X4w1EdgXbUkUMjg4NzIyMDE0LDEyMzE4ODYyOTY=", "ïƒ¿Xow1EdkXbUkjRDpcRGV2XGxpYlxTUUxpdGUzXGV4ZVxUZXN0U1FMMy5leGU="]}'
It matches the {\f1\fs20 var / const / out} parameters declaration of the method:
! function ComplexCall(const Ints: TIntegerDynArray; Strs1: TRawUTF8DynArray;
!   var Str2: TWideStringDynArray; const Rec1: TVirtualTableModuleProperties;
!   var Rec2: TSQLRestCacheEntryValue): TSQLRestCacheEntryValue;
And its implementation:
!function TServiceCalculator.ComplexCall(const Ints: TIntegerDynArray;
!  Strs1: TRawUTF8DynArray; var Str2: TWideStringDynArray; const Rec1: TVirtualTableModuleProperties;
!  var Rec2: TSQLRestCacheEntryValue): TSQLRestCacheEntryValue;
!var i: integer;
!begin
!  result := Rec2;
!  result.JSON := StringToUTF8(Rec1.FileExtension);
!  i := length(Str2);
!  SetLength(Str2,i+1);
!  Str2[i] := UTF8ToWideString(RawUTF8ArrayToCSV(Strs1));
!  inc(Rec2.ID);
!  dec(Rec2.Timestamp);
!  Rec2.JSON := IntegerDynArrayToCSV(Ints,length(Ints));
!end;
Note that {\f1\fs20 TIntegerDynArray}, {\f1\fs20 TRawUTF8DynArray} and {\f1\fs20 TWideStringDynArray} values were marshaled as JSON arrays, whereas complex records (like {\f1\fs20 TSQLRestCacheEntryValue}) have been @*Base64@ encoded.
If you want to transmit some binary blob content, consider using a {\f1\fs20 @*RawByteString@} kind of parameter, which will transmit a Base64-encoded JSON text on the wire.
The framework is able to handle class instances as parameters, for instance with the following interface, using a {\f1\fs20 TPersistent} child class with published properties (it will be the same for {\f1\fs20 @*TSQLRecord@} @*ORM@ instances):
!type
!  TComplexNumber = class(TPersistent)
!  private
!    fReal: Double;
!    fImaginary: Double;
!  public
!    constructor Create(aReal, aImaginary: double); reintroduce;
!  published
!    property Real: Double read fReal write fReal;
!    property Imaginary: Double read fImaginary write fImaginary;
!  end;
!
!  IComplexCalculator = interface(ICalculator)
!    ['{8D0F3839-056B-4488-A616-986CF8D4DEB7}']
!    /// purpose of this unique method is to substract two complex numbers
!    // - using class instances as parameters
!    procedure Substract(n1,n2: TComplexNumber; out Result: TComplexNumber);
!  end;
As stated above, it is not possible to return a class as a result of a {\f1\fs20 function} (who will be responsible of handling its life-time?). So in this method declaration, the result is declared as {\f1\fs20 out} parameter.
During the transmission, published properties of {\f1\fs20 TComplexNumber} parameters will be serialized as standard JSON objects within the {\f1\fs20 "result":[...]} JSON array:
$ POST root/ComplexCalculator.Substract
$ (...)
$ [{"Real":2,"Imaginary":3},{"Real":20,"Imaginary":30}]
will be answered as such:
$ {"result":[{"Real":-18,"Imaginary":-27}]}
:      Returning a JSON object
Note that if {\f1\fs20 TServiceFactoryServer.ResultAsJSONObject} is set to {\f1\fs20 true}, the outgoing values won't be emitted within a {\f1\fs20 "result":[...]} JSON array, but via a {\f1\fs20 "result":\{... \}} JSON object, with the {\f1\fs20 var/out} parameter names as object fields, and {\f1\fs20 "Result":} for a function result:
$ {"result":{"Result":{"Real":-18,"Imaginary":-27}}}
The {\f1\fs20 TServiceFactoryServer.ResultAsJSONObjectWithoutResult} property may be used to avoid the main {\f1\fs20 "Result":} object.
Instead of this JSON array content, returned by default:
!GET root/Calculator/Add?n1=1&n2=2
! ...
!{"result":[3]}
The following JSON will be returned if {\f1\fs20 TServiceFactoryServer.ResultAsJSONObject} is {\f1\fs20 true}:
!GET root/Calculator/Add?n1=1&n2=2
! ...
!{"result":{"Result":3}}
Or, if {\f1\fs20 TServiceFactoryServer.ResultAsJSONObjectWithoutResult} is {\f1\fs20 true}:
!GET root/Calculator/Add?n1=1&n2=2
! ...
!{"Result":3}
All those JSON array or object contents fulfill perfectly standard JSON declarations, so can be generated and consumed directly by any @*AJAX@ client. The {\f1\fs20 TServiceFactoryServer. ResultAsJSONObject} option make it even easier to consume {\i mORMot} services, since all outgoing values will be named in the {\f1\fs20 "result":} JSON object.
:     Returning raw JSON content
By default, if you want to transmit a JSON content with interface-based services, using a {\f1\fs20 RawUTF8} will convert it to a JSON string. Therefore, any JSON special characters (like {\f1\fs20 "} or {\f1\fs20 \\} or {\f1\fs20 [}) will be escaped. This will slow down the process on both server and client side, and increase transmission bandwidth.
For instance, if you define such a method:
!function TServiceRemoteSQL.Execute(const aSQL: RawUTF8; aExpectResults, aExpanded: Boolean): RawUTF8;
!var res: ISQLDBRows;
!begin
!  if fProps=nil then
!    raise Exception.Create('Connect call required before Execute');
!  res := fProps.ExecuteInlined(aSQL,aExpectResults);
!  if res=nil then
!    result := '' else
!!    result := res.FetchAllAsJSON(aExpanded);
!end;
The {\f1\fs20 FetchAllAsJSON()} method will return a JSON array content, but will be escaped as a JSON string when transmitted via a {\f1\fs20 RawUTF8} variable.
A dedicated {\f1\fs20 @**RawJSON@} type has been defined, and will specify to the {\f1\fs20 mORMot} core that the @*UTF-8@ text is a valid JSON content, and should not be escaped.
That is, defining the method as followed will increase process speed and reduce used bandwidth:
!function TServiceRemoteSQL.Execute(const aSQL: RawUTF8; aExpectResults, aExpanded: Boolean): RawJSON;
See sample "{\i 16 - Execute SQL via services}" for some working code using this feature.
As a consequence, using {\f1\fs20 RawJSON} will also make the transmitted content much more @*AJAX@ friendly, since the returned value will be a valid JSON array or object, and not a JSON string which will need {\i JavaScript} "unstringification".
:     Returning errors
In case of an error, the standard message object will be returned:
${
$ "ErrorCode":400,
$ "ErrorText":"Error description"
$}
The following error descriptions may be returned by the service implementation from the server side:
|%35%65
|\b {\f1\fs20 ErrorText}|Description\b0
|Method name required|{\f1\fs20 TSQLRestRoutingJSON_RPC} call without {\f1\fs20 "method":} field
|Unknown method|{\f1\fs20 TSQLRestRoutingJSON_RPC} call with invalid method name (in this mode, there is no specific message, since a JSON answer {\i may} be a valid request)
|Parameters required|The server expect at least a void JSON array (aka {\f1\fs20 []}) as parameters
|Unauthorized method|This method is not allowed with the current authenticated user group - see {\i @*Security@} above
|Not allowed to publish signature|The client requested the interface signature, but this has not been allowed on the server policy (see {\f1\fs20 TServiceContainerServer. PublishSignature})
|... instance id:? not found or deprecated|The supplied {\f1\fs20 "id":} parameter points to a wrong instance (in {\f1\fs20 sicPerSession / sicPerUser / sicPerGroup} mode)
|ExceptionClass: Exception Message (with {\f1\fs20 500 Internal Server Error})|An exception was raised during method execution
|%
On the client side, you may encounter the following {\f1\fs20 EInterfaceFactoryException} messages, starting with the generic '{\f1\fs20 Invalid fake IInterfaceName.MethodName interface call}' text:
|%35%65
|\b {\f1\fs20 ErrorText}|Description\b0
|unexpected self|{\f1\fs20 self} does exist as low-level implementation detail, but is not intended to be transmitted
|JSON array/object result expected|content returned from the Server was neither a JSON array nor a JSON object
|unexpected parameter "...."|the Server returned a JSON object with an unknown or invalid member name
|returned object record variant array RawJSON|a returned {\f1\fs20 class}, record, variant, dynamic array of {\f1\fs20 RawJSON} value was not properly serialized
|missing or invalid value|a returned string or numerical value is not valid JSON content
|%
:    Returning content as XML
By default, interface-based services of a {\i mORMot} server will always return a JSON array. But you may  (or a JSON object, if {\f1\fs20 TServiceFactoryServer.ResultAsJSONObject} or {\f1\fs20 ResultAsJSONObjectWithoutResult} is {\f1\fs20 true}).
With some kind of clients (e.g. if they are made by a third party), it could be useful to return @**XML@ content instead.
Your {\i mORMot} server is able to let its interface-based services return XML context instead, or in addition to the default JSON format.
:     Always return XML content
If you want all methods of a given {\f1\fs20 interface} to return XML content instead of JSON, you can set {\f1\fs20 TServiceFactoryServer.ResultAsXMLObject} to {\f1\fs20 true}.
Instead of this JSON array content, returned by default:
!GET root/Calculator/Add?n1=1&n2=2
! ...
!{"result":[3]}
The following XML will be returned if {\f1\fs20 TServiceFactoryServer.ResultAsXMLObject} is {\f1\fs20 true}:
!GET root/Calculator/Add?n1=1&n2=2
! ...
$$<?xml version="1.0" encoding="UTF-8"?>
$$<result><Result>3</Result></result>
Conversion is processed from the JSON content generated by the {\i mORMot} kernel, via a call to {\f1\fs20 JSONBufferToXML()} function, which performs the XML generation without almost no memory allocation. So only a slightly performance penalty may be noticed (much faster in practice than most node-based XML producers available).
One drawback of using this {\f1\fs20 TServiceFactoryServer.ResultAsXMLObject} property is that your regular {\i Delphi} or AJAX clients won't be able to consume the service any more, since they expect JSON content.\line If you want your service to be consumed either by XML and JSON, you will need two services. You can therefore define a dedicated {\f1\fs20 interface} to return XML, and then register this interface to return only XML:
!type
!  ICalculator = interface(IInvokable)
!    ['{9A60C8ED-CEB2-4E09-87D4-4A16F496E5FE}']
!    /// add two signed 32-bit integers
!    function Add(n1,n2: integer): integer;
!  end;
!!  ICalculatorXML = interface(ICalculator)
!!    ['{0D682D65-CE0F-441B-B4EC-2AC75E357EFE}']
!!  end; // no additional method, just new name and GUID
!
!!  TServiceCalculator = class(TInterfacedObject, ICalculator,ICalculatorXML)
!  public // implementation class should implement both interfaces
!    function Add(n1,n2: integer): integer;
!  end;
!
!...
!  aServer.ServiceRegister(TServiceCalculator,[TypeInfo(ICalculator)],sicShared);
!!  aServer.ServiceRegister(TServiceCalculator,[TypeInfo(ICalculatorXML)],sicShared).
!!    ResultAsXMLObject := True;
!...
There will therefore be two running service instances (e.g. here two instances of {\f1\fs20 TServiceCalculator}, one for {\f1\fs20 ICalculator} and one for {\f1\fs20 ICalculatorXML}). It could be an issue, in some cases.
And such a dedicated interface may need more testing and code on the server side, since they will be accessible from two URIs:
!GET root/Calculator/Add?n1=1&n2=2
! ...
!{"result":{"Result":3}}
and for {\f1\fs20 ICalculatorXML interface}:
!GET root/CalculatorXML/Add?n1=1&n2=2
! ...
$$<?xml version="1.0" encoding="UTF-8"?>
$$<result><Result>3</Result></result>
:     Return XML content on demand
As an alternative, you can let the {\i mORMot} server inspect the incoming HTTP headers, and return the content as XML if the "{\f1\fs20 Accept: }" header is exactly "{\f1\fs20 application/xml}" or "{\f1\fs20 text/xml}".
You can set the {\f1\fs20 TServiceFactoryServer.ResultAsXMLObjectIfAcceptOnlyXML} property to enable this HTTP header detection:
!  aServer.ServiceRegister(TServiceCalculator,[TypeInfo(ICalculator)],sicShared).
!!    ResultAsXMLObjectIfAcceptOnlyXML := true;
For standard requests, the incoming HTTP header will be either void, either "{\f1\fs20 Accept: */*}", so will return JSON content.\line But if the client set either "{\f1\fs20 Accept: application/xml}" or "{\f1\fs20 Accept: text/xml}" in its header, then it will return an XML document.
Instead of this JSON content:
!GET root/Calculator/Add?n1=1&n2=2
!Accept: */*
! ...
!{"result":{"Result":3}}
The following XML will be returned:
!GET root/Calculator/Add?n1=1&n2=2
!!Accept: application/xml
! ...
$$<?xml version="1.0" encoding="UTF-8"?>
$$<result><Result>3</Result></result>
as it will with "{\f1\fs20 text/xml}":
!GET root/Calculator/Add?n1=1&n2=2
!!Accept: text/xml
! ...
$$<?xml version="1.0" encoding="UTF-8"?>
$$<result><Result>3</Result></result>
Note that the header is expected to be "{\f1\fs20 Accept: application/xml}" or "{\f1\fs20 Accept: text/xml}" as {\i exact value}.\line For instance "{\f1\fs20 Accept: text/html,application/xml,*/*}" won't be detected by the server, and will return regular JSON:
!GET root/Calculator/Add?n1=1&n2=2
!Accept: text/html,application/xml,*/*
! ...
!{"result":{"Result":3}}
Your XML client should therefore be able to force the exact content of the HTTP "{\f1\fs20 Accept:}" header.
Together with parameter values optionally encoded at URI level - available with {\f1\fs20 TSQLRestRoutingREST} default routing scheme (see {\f1\fs20 ?n1=1&n2=2} above)- it could be an useful alternative to consume {\i mORMot} services from any XML-based client.
:    Custom returned content
Note that even if the response format is a JSON object by default, and expected as such by our {\f1\fs20 TServiceContainerClient} implementation, there is a way of returning any content from a remote request.
It may be used by @*AJAX@ or HTML applications to return any kind of data, i.e. not only JSON results, but pure text, HTML or even binary content. Our {\f1\fs20 TServiceFactoryClient} instance is also able to handle such requests, and will save client-server bandwidth when transmitting some BLOB data (since it won't serialized the content with {\f1\fs20 Base64} encoding).
In order to specify a custom format, you can use the following {\f1\fs20 @*TServiceCustomAnswer@ record} type as the {\f1\fs20 result} of an {\f1\fs20 interface function}:
!  TServiceCustomAnswer = record
!    Header: RawUTF8;
!    Content: RawByteString;
!    Status: cardinal;
!  end;
The {\f1\fs20 Header} field shall be not null (i.e. not equal to ''), and contains the expected content type header (e.g. {\f1\fs20 TEXT_CONTENT_TYPE_HEADER} or {\f1\fs20 HTML_CONTENT_TYPE_HEADER}).\line Then the {\f1\fs20 Content} value will be transmitted back directly to the client, with no JSON @*serialization@. Of course, no {\f1\fs20 var} nor {\f1\fs20 out} parameter will be transmitted (since there is no JSON result array any more).\line Finally, the {\f1\fs20 Status} field could be overridden with a property HTML code, if the default {\f1\fs20 HTTP_SUCCESS} is not enough for your purpose. Note that when consumed from {\i Delphi} clients, {\f1\fs20 HTTP_SUCCESS} is expected to be returned by the server: you should customize {\f1\fs20 Status} field only for plain AJAX / web clients.
In order to implement such method, you may define such an interface:
!  IComplexCalculator = interface(ICalculator)
!    ['{8D0F3839-056B-4488-A616-986CF8D4DEB7}']
!    function TestBlob(n: TComplexNumber): TServiceCustomAnswer;
!  end;
This may be implemented for instance as such:
!function TServiceComplexCalculator.TestBlob(n: TComplexNumber): TServiceCustomAnswer;
!begin
!  Result.Header := TEXT_CONTENT_TYPE_HEADER;
!  Result.Content := FormatUTF8('%,%',[n.Real,n.Imaginary]);
!  // leave Result.Header to its default value
!end;
This will return not a JSON object, but a plain TEXT content.
Regression tests will make the following process:
!  with CC.TestBlob(C3) do begin
!    Check(Header=TEXT_CONTENT_TYPE_HEADER);
!    Check(Content=FormatUTF8('%,%',[C3.Real,C3.Imaginary]));
!  end;
Note that since there is only one BLOB content returned, no {\f1\fs20 var} nor {\f1\fs20 out} parameters are allowed to be defined for this method. If this is the case, an exception will be raised during the {\f1\fs20 interface} registration step. But you can define any {\f1\fs20 const} parameter needed, to specify your request.
You may also be able to use this feature to implement custom @*UTF-8@ HTML creation, setting the {\f1\fs20 Header} value to {\f1\fs20 HTML_CONTENT_TYPE_HEADER} constant, and using our fast @81@ for the rendering.\line Remember that in {\f1\fs20 TSQLRestRoutingJSON} mode, you can encode any simple parameter value at URI level, to transmit your browsing context.
\page
:65 Comparison with WCF
Microsoft {\i Windows Communication Foundation} is the unified programming model provided by Microsoft for building service-oriented applications.\line See @http://msdn.microsoft.com/en-us/library/dd456779
Here is a short reference table of @**WCF@ / {\i mORMot} @*SOA@ features and implementation of the @*REST@ful pattern.
|%30%35%35
|\b Feature|WCF|mORMot\b0
|Internal design|@*SOAP@ with REST integration|RESTful
|Hosting|exe/service/ISS/WAS|in-process/exe/service
|Scalability/balancing|up to WAS|by dedicated hosting
|MetaData|WSDL+XML|JSON contract
|Service contract|{\f1\fs20 interface}|{\f1\fs20 interface}
|Data contract|{\f1\fs20 class}|{\f1\fs20 class/record}
|ORM integration|separated|integrated in the model
|URI definition|attribute-driven|REST/@*JSON-RPC@ convention-driven, or class-driven
|Service contract|{\f1\fs20 interface} + attributes|{\f1\fs20 interface} + shared Model
|Versioning|XML name-space|{\f1\fs20 interface} signature
|Message protocol|SOAP/custom|RESTful
|Messaging|single/duplex|stateless (like HTTP)
|Sequence|attributes on methods|{\f1\fs20 interface} life time
|Transactional|fully transactional|on implementation side
|Instance life time|per call, per session, single|per call, per session, per user, per group, per thread, single, client-driven
|Configuration|{\f1\fs20 .config} file or code|{\i @*convention over configuration@}\line optionally tuned by code
|Client acccess|Layer source should be generated|No layer, but direct registration
|End points|One end-point per contract|Unique or shared end-point
|Operation|synchronous/asynchronous|synchronous (REST)
|Session|available (optional)|available (optional)
|Encryption|at Service level|at communication level
|Compression|at Service level|at communication level
|Serialization|XML/binary/JSON|JSON/XML/custom
|Communication protocol|HTTP/HTTPS/TCP/pipe/MSMQ|HTTP/HTTPS/TCP/pipe/messages/in-process
|HTTP/HTTPS server|{\i @*http.sys@}|{\i http.sys}/native (winsock)
|Authentication|Windows or custom|Windows, ORM-based, or class-driven
|Authorization|by attribute or config files|per user group, or class-driven
|Threading|by attributes|at service/method level
|Weight|middle (GC, JIT, .dll)|low
|Speed|good|high
|Extensibility|verbose but complete|customizable
|Standard|de facto|@*KISS@ design (e.g. JSON, HTTP)
|Source code|closed|published
|License|proprietary|Open
|Price|depends|Free
|Support|official + community|Synopse + community
|Runtime required|.Net framework (+ISS/WAS)|none (blank OS)
|%
About instance life time, note that in WCF {\f1\fs20 InstanceContextMode.Single} is in fact the same as {\f1\fs20 sicShared} within {\i mORMot} context: only one instance is used for all incoming calls and is not recycled subsequent to the calls. Therefore, {\f1\fs20 sicSingle} mode (which is {\i mORMot}'s default) maps {\f1\fs20 InstanceContextMode.PerCall} in WCF, meaning that one instance is used per call.
We may be tempted to say that {\i mORMot} SOA architecture is almost complete, even for a young and {\i Open Source} project. Some features (like {\i per user}, {\i per group} or {\i client-driven} instance life time, or Windows Messages local communication) are even unique to {\i mORMot}. In fact, {\f1\fs20 sicClientDriven} is pretty convenient when implementing a Service Oriented Architecture.
Of course, WCF features its @**SOAP@-based architecture. But WCF also suffers from it: due to this ground-up message design, it will always endure its SOAP overweight, which is "Simple" only by name, not by reputation.
If you need to communicate with an external service provider, you can easily create a SOAP @*gateway@ from {\i Delphi}, as such:
- Import the WSDL (Web Service Definition Language) definition of a web service and turn it into a {\i Delphi} import unit;
- Publish the interface as a {\i mORMot} server-side implementation class.
Since SOAP features a lot of requirements, and expects some plumping according to its format (especially when services are provided from C# or Java), we choose to not re-invent the wheel this time, and rely on existing {\i Delphi} libraries (available within the {\i Delphi} IDE) for this purpose. If you need a cross-platform SOAP 1.1 compatible solution, or if you version of {\i Delphi} does not include SOAP process, you may take a look at @http://wiki.freepascal.org/Web_Service_Toolkit which is a web services package for FPC, Lazarus and {\i Delphi}.
But for service communication within the {\i mORMot} application domain, the RESTful / JSON approach gives much better performance and ease of use. You do not have to play with WSDL or unit wrappers, just share some {\f1\fs20 interface} definition between clients and servers. Once you have used the {\f1\fs20 ServiceRegister()} or {\f1\fs20 ServiceDefine()} methods of {\i mORMot}, you will find out how the WCF plumbing is over-sized and over-complicated: imagine that WCF allows only one end-point per interface/contract - in a @47@ world, where {\i interface segregation} should reign, it is not the easier way to go!
Optionally, {\i mORMot}'s interface based services allow to publish their result as XML, and encode the incoming parameters at URI level. It makes it a good alternative to SOAP, in the XML world.
At this time, the only missing feature of {\i mORMot}'s SOA is transactional process, which must be handled on server side, within the service implementation (e.g. with explicit commit or rollback).
;{\i @*Event Sourcing@} and @*Unit Of Work@ design patterns have been added to the {\i mORMot} official road map, in order to handle @*transaction@s on the SOA side, relying on ORM for its data persistence, but not depending on database transactional abilities. In fact, transactions should better be implemented at SOA level, as we do want transactions to be database agnostic ({\i @*SQLite3@} has a limited per-connection transactional scheme, and we do not want to rely on the DB layer for this feature). {\i Event Sourcing} sounds to be a nice pattern to implement a strong and efficient transactional process in our framework - see @http://bliki.abdullin.com/event-sourcing/why
:86Cross-Platform clients
%cartoon07.png
Current version of the main framework units target only {\i Win32} / {\i Win64} systems under Delphi, and (in a preliminary state) {\i Windows} or {\i @*Linux@} under FPC.\line It allows to make easy self-hosting of {\i mORMot} servers for local business applications in any corporation, or pay cheap hosting in the @*Cloud@, since {\i mORMot} CPU and RAM expectations are much lower than a regular {\f1\fs20 IIS-WCF-MSSQL-.Net} stack.\line But in a @17@, you will probably need to create clients for platforms outside the support platform sets world, especially mobile devices or AJAX applications.
A set of @**cross-platform@ client units is therefore available in the {\f1\fs20 CrossPlatform} sub-folder of the source code repository. It allows writing any client in modern {\i object pascal} language, for:
- Any version of {\i Delphi}, on any platform ({\i Mac @*OSX@}, or any mobile supported devices);
- {\i @*FreePascal@} Compiler (in 2.6.4, 2.7.1 or 3.x branches - preferred is {\i 3.2 fixes});
- {\i @*Smart Mobile Studio@} (2.1 and up), to create AJAX or mobile applications (via {\i @*PhoneGap@}, if needed).
The units are the following:
|%37%63
|\b Unit Name|Description\b0
|{\f1\fs20 SynCrossPlatformREST.pas}|Main unit, implementing secured ORM and SOA @*REST@ful client
|{\f1\fs20 SynCrossPlatformCrypto.pas}|@*SHA256@ and @*crc32@ algorithms, used for authentication
|{\f1\fs20 SynCrossPlatformJSON.pas}|Optimized JSON process (not used by {\i Smart})
|{\f1\fs20 SynCrossPlatformSpecific.pas}|System-specific functions, e.g. HTTP clients
|%
\page
This set of units will provide a solid and shared ground for the any kind of clients:
- Connection to a {\i mORMot} server via HTTP, with full REST support;
- Support of weak or default authentication to secure the transfer - see @18@;
- Definition of the {\f1\fs20 TSQLRecord} class, using RTTI when available on {\i Delphi} or {\i FreePascal}, and generated code for {\i Smart Mobile Studio};
- Remote @*CRUD@ operations, via @*JSON@ and @*REST@, with a {\f1\fs20 TSQLRestClientURI} class, with the same methods as with the {\f1\fs20 mORMot.pas} framework unit;
- Optimized {\f1\fs20 TSQLTableJSON} class to handle a JSON result table, as returned by {\i mORMot}'s REST server ORM - see @87@;
- @*Batch@ process - see @28@ - for transactional and high-speed writes;
- @49@ with parameters marshalling;
- @63@ with parameters marshalling and instance-life time;
- Mapping of most supported field types, including e.g. @*ISO 8601@ date/time encoding, @*BLOB@s and {\f1\fs20 @*TModTime@}/{\f1\fs20 @*TCreateTime@} - see @26@;
- Complex {\f1\fs20 record} types are also exported and consumed via JSON, on all platforms (for both ORM and SOA methods);
- Integrated debugging methods, used by both ORM and SOA process, able to log into a local file or to a remote server - see @103@;
- Some cross-platform low-level functions and types definitions, to help share as much code as possible between your projects.
In the future, C# or Java clients may be written.\line The {\f1\fs20 CrossPlatform} sub-folder code could be used as reference, to write minimal and efficient clients on any platform. Our REST model is pretty straightforward and standard, and use of JSON tends to leverage a lot of potential marshalling issues which may occur with XML or binary formats.
In practice, a code generator embedded in the {\i mORMot} server can be used to create the client wrappers, using the @81@ included on the server side. With a click, you can generate and download a client source file for any supported platform. A set of {\f1\fs20 .mustache} templates is available, and can be customized or extended to support any new platform: any help is welcome, especially for targeting Java or C# clients.
\page
: Available client platforms
:69  Delphi FMX / FreePascal FCL cross-platform support
Latest versions of {\i Delphi} include the {\i @**FireMonkey@} (FMX) framework, able to deliver multi-device, true native applications for {\i Windows}, {\i Mac @*OSX@}, {\i @*Android@} and {\i iOS} ({\i @*iPhone@}/{\i @*iPad@}).\line Our {\f1\fs20 SynCrossPlatform*} units are able to easily create clients for those platforms.
Similarly, these units can be compiled with @*FreePascal@, so that any {\i mORMot} server could be consumed from the numerous supported platforms of this compiler.
In order to use those units, ensure in your IDE that the {\f1\fs20 CrossPlatform} sub-folder of the {\i mORMot} source code repository is defined in your {\i Library Search Path}.
:   Cross-platform JSON
We developed our own cross-platform @*JSON@ process unit in {\f1\fs20 SynCrossPlatformJSON.pas}, shared with {\i Delphi} and {\i FreePascal}.\line In fact, it appears to be easier to use (since it is {\f1\fs20 variant}-based and with {\i late-binding} abilities) and run much faster than the official {\f1\fs20 DBXJSON.pas} unit shipped with latest versions of {\i Delphi}, as stated by the "{\f1\fs20 25 - JSON performance}" sample:
$ 2.2. Table content:
$- Synopse crossplatform: 41,135 assertions passed  20.56ms  400,048/s  1.9 MB
$- DBXJSON: 41,136 assertions passed  240.84ms  34,159/s  9.9 MB
Our {\f1\fs20 TSQLTableJSON} class is more than 10 times faster than standard {\f1\fs20 DBXJSON} unit, when processing a list of results as returned by a {\i mORMot} server.\line The latest value on each line above is the memory consumption. It should be of high interest on mobile platforms, where memory allocation tends to be much slower and sensitive than on Windows (where the {\i @*FastMM4@} memory manager does wonders). Our unit consumes 5 times less memory than the RTL's version.
We did not include {\i XSuperObject} here, which is cross-platform, but performs even worse than {\f1\fs20 DBXJSON} in terms of speed. Other libraries - as {\i SuperObject} or {\i dwsJSON} - are not cross-platform.\line See @http://blog.synopse.info/post/json-benchmark-delphi-mormot-superobject-dwsjson-dbxjson for details about this comparison.
A special mention is due to {\i dwsJSON}, which performs very well, but only on Windows, and is slower than {\i mORMot}'s implementation:
$- Synopse ORM loop: 41,135 assertions passed  6.18ms  1,330,153/s  1.1 MB
$- Synopse ORM list: 41,135 assertions passed  6.47ms  1,270,775/s  952 KB
$- Synopse crossplatform: 41,135 assertions passed  20.56ms  400,048/s  1.9 MB
$- Super object properties: 41,136 assertions passed  2.20s  3,739/s  6.3 MB
$- dwsJSON: 41,136 assertions passed  32.05ms  256,628/s  4.7 MB
$- DBXJSON: 41,136 assertions passed  240.84ms  34,159/s  9.9 MB
The "{\f1\fs20 Synopse ORM}" lines stand for the {\f1\fs20 TSQLTableJSON} class as implemented in {\f1\fs20 mORMot.pas}. It uses our optimized @*UTF-8@ functions and classes, in-place escaping together with our {\f1\fs20 @*RawUTF8@} custom string type as implemented in {\f1\fs20 SynCommons.pas}, so that it is 3 times faster than our cross-platform units, and 40 times than {\f1\fs20 DBXJSON}, using much less memory. Some tricks used by {\f1\fs20 Synopse ORM} rely on pointers and are not compatible with the {\i NextGen} compiler or the official {\i Delphi} road-map, so the {\f1\fs20 Synopse crossplatform} uses diverse algorithm, but offers still pretty good performance.
This unit features a {\f1\fs20 TJSONVariantData} custom variant type, similar to @80@, available in the main {\i mORMot} framework.\line It allows writing such nice and readable code, with late-binding:
!var doc: variant;
!    json,json2: string;
!...
!  doc := JSONVariant('{"test":1234,"name":"Joh\\"n\\r","zero":0.0}');
!  assert(doc.test=1234);
!  assert(doc.name='Joh"n'#13);
!  assert(doc.name2=null);
!  assert(doc.zero=0);
!  json := doc; // conversion to JSON text when assigned to a string variable
!  assert(json='{"test":1234,"name":"Joh\\"n\\r","zero":0}');
!  doc.name2 := 3.1415926;
!  doc.name := 'John';
!  json := doc;
!  assert(json='{"test":1234,"name":"John","zero":0,"name2":3.1415926}');
The unit is also able to serialize any {\f1\fs20 TPersistent} class, i.e. all published properties could be written or read from a JSON object representation. It also handles nested objects, stored as {\f1\fs20 TCollection}.\line See for instance in the {\f1\fs20 SynCrossPlatformTests} unit:
!type
!  TMainNested = class(TCollectionItem)
!  private
!    fNumber: double;
!    fIdent: RawUTF8;
!  published
!    property Ident: RawUTF8 read fIdent write fIdent;
!    property Number: double read fNumber write fNumber;
!  end;
!
!  TMain = class(TPersistent)
!  private
!    fName: RawUTF8;
!    fNested: TCollection;
!    fList: TStringList;
!  public
!    constructor Create;
!    destructor Destroy; override;
!  published
!    property Name: RawUTF8 read fName write fName;
!    property Nested: TCollection read fNested;
!    property List: TStringList read fList;
!  end;
!
!  obj1 := TMain.Create;
!  obj2 := TMain.Create;
!...
!  obj1.Name := IntToStr(i);
!  item := obj1.Nested.Add as TMainNested;
!  item.Ident := obj1.Name;
!  item.Number := i/2;
!  obj1.list.Add(obj1.Name);
!  json := ObjectToJSON(obj1);
!  if i=1 then
!    assert(json='{"Name":"1","Nested":[{"Ident":"1","Number":0.5}],"List":["1"]}');
!  JSONToObject(obj2,json);
!  assert(obj2.Nested.Count=i);
!  json2 := ObjectToJSON(obj2);
!  assert(json2=json);
!...
Of course, this serialization feature is used for the {\f1\fs20 TSQLRecord} ORM class.
Due to lack of RTTI, {\f1\fs20 record} serialization is supported via some functions generated by the server with the code wrappers.
:   Delphi OSX and NextGen
In order to be compliant with the {\i NextGen} revision, our {\f1\fs20 SynCrossPlatform*} units follow the expectations of this new family of cross-compilers, which targets {\i @*Android@} and {\i @*iOS@}.\line In particular, we rely only on the {\f1\fs20 string} type for text process and storage, even at JSON level, and we tried to make object allocation ARC-compatible. Some types have been defined, e.g. {\f1\fs20 THttpBody}, {\f1\fs20 TUTF8Buffer} or {\f1\fs20 AnsiChar}, to ensure that our units will compile on all supported platforms.
On {\i Delphi}, the {\i @*Indy@} library is used for HTTP requests. It is cross-platform by nature, so should work on any supported system. For SSL support with {\i iOS} and {\i Android} clients, please follow instructions at @http://blog.marcocantu.com/blog/using_ssl_delphi_ios.html you may also download the needed {\f1\fs20 libcrypto.a} and {\f1\fs20 libssl.a} files from @http://indy.fulgan.com/SSL/OpenSSLStaticLibs.7z
Feedback is needed for the mobile targets, via FMX.\line In fact, we rely for our own projects on {\i @*Smart Mobile Studio@} for our mobile applications, so the {\i Synopse} team did not test {\i Delphi NextGen} platforms (i.e. {\i iOS} and {\i Android}) as deep as other systems. Your input will be very valuable and welcome, here!
:   FreePascal clients
{\f1\fs20 SynCrossPlatform*} units support the {\i @**FreePascal@} Compiler, in its 2.7.1 / 3.x branches.\line Most of the code is shared with {\i Delphi}, including RTTI support and all supported types.
Some restrictions apply, though.
Due to a bug in {\i FreePascal} implementation of {\f1\fs20 variant} late binding, the following code won't work as expected on older revisions of FPC:
!  doc.name2 := 3.1415926;
!  doc.name := 'John';
Under oldest {\i FreePascal}, you have to write:
!  TJSONVariantData(doc)['name2'] := 3.1415926;
!  TJSONVariantData(doc)['name'] := 'John';
In fact, the way late-binding properties are implemented in the {\i FreePascal} in some fully compatible with {\i Delphi} expectations. The {\i FreePascal} maintainers did some initial fix (the {\f1\fs20 variant} instance is now passed by reference), so above code seems to work on current FPC trunk.
As a result, direct access to {\f1\fs20 TJSONVariantData} instances, and not a {\f1\fs20 variant} variable, may be both safer and faster when using FPC.
In the Lazarus IDE, we also observed that the debugger is not able to handle our custom {\f1\fs20 variant} type. If you look at any {\f1\fs20 TJSONVariantData} instance with the debugger, an error message "{\i unsupported variant type}" will appear. As far as we found out, this is a Lazarus limitation. Delphi, on its side, is able to display any custom {\f1\fs20 variant} type in its debugger, after conversion to {\f1\fs20 string}, i.e. its JSON representation.
Another issue with the 2.7.1 / 3.1.1 revisions is how the new {\f1\fs20 string} type is implemented.\line In fact, if you use a string variable containing an @*UTF-8@ encoded text, then the following line will reset the result code page to the system code page:
!function StringToJSON(const Text: string): string;
!  ...
!  result := '"'+copy(Text,1,j-1); // here FPC 2.7.1 erases UTF-8 encoding
!  ...
It sounds like if {\f1\fs20 '"'} will force the code page of {\f1\fs20 result} to be not an UTF-8 content.\line With {\i Delphi}, this kind of statements work as expected, even for {\f1\fs20 AnsiString} values, and {\f1\fs20 '"'} constant is handled as {\f1\fs20 RawByteString}. We were not able to find an easy and safe workaround for FPC yet. Input is welcome in this area, from any expert!
You have to take care of this limitation, if you target the {\i Windows} operating system with FPC (and {\i @*Lazarus@}). Under other systems, the default code page is likely to be UTF-8, so in this case our {\f1\fs20 SynCrossPlatform*} units will work as expected.
We found out the {\i FreePascal} compiler to work very well, and result in small and fast executables. For most common work, timing is comparable with {\i Delphi}. The memory manager is less optimized than {\i @*FastMM4@} for rough simple threaded tests, but is cross-platform and designed to be more efficient in multi-thread mode: in fact, it has no giant lock, as {\i FastMM4} suffers.
:105   Local or remote logging
You can use the {\f1\fs20 TSQLRest.Log()} overloaded methods to log any content into a file or a remote server.
All ORM and SOA functions of the {\f1\fs20 TSQLRest} instance will create the expected log, just with the main {\i mORMot} units running on Win32/Win64 - see @104@.\line For instance, here are some log entries created during the {\f1\fs20 RegressionTest.dpr} process:
$16:47:15 Trace  POST root/People status=201 state=847 in=92 out=0
$16:47:15 DB  People.ID=200 created from {"FirstName":"First200","LastName":"Last200","YearOfBirth":2000,"YearOfDeath":2025,"Sexe":0}
$16:47:15 SQL  select RowID,FirstName,LastName,YearOfBirth,YearOfDeath,Sexe from People
$16:47:15 Trace  GET root?sql=select+RowID%2CFirstName%2CLastName%2CYearOfBirth%2CYearOfDeath%2CSexe+from+People status=200 state=847 in=0 out=21078
$16:47:15 SQL  select RowID,YearOfBirth,YearOfDeath from People
$16:47:15 Trace  GET root?sql=select+RowID%2CYearOfBirth%2CYearOfDeath+from+People status=200 state=847 in=0 out=10694
$16:47:15 SQL  select RowID,FirstName,LastName,YearOfBirth,YearOfDeath,Sexe from People where yearofbirth=:(1900):
$16:47:15 Trace  GET root?sql=select+RowID%2CFirstName%2CLastName%2CYearOfBirth%2CYearOfDeath%2CSexe+from+People+where+yearofbirth%3D%3A%281900%29%3A status=200 state=847 in=0 out=107
$16:47:15 Trace  DELETE root/People/16 status=200 state=848 in=0 out=0
$16:47:15 DB  Delete People.ID=16
Then, our {\i Log View} tool is able to run as a remote log server, and display the incoming events in real-time - see @103@.\line Having such logs available will be pretty convenient, especially when debugging applications on a mobile device, or a remote computer.
:90  Smart Mobile Studio support
{\i @**Smart Mobile Studio@} - see @http://www.smartmobilestudio.com - is a complete RAD environment for writing cutting edge HTML5 mobile applications. It ships with a fully fledged compiler capable of compiling {\i Object Pascal} (in a modern dialect call {\i @*SmartPascal@}) into highly optimized and raw {\i @*JavaScript@}.
There are several solutions able to compile to {\i JavaScript}.\line In fact, we can find several families of compilers:
- {\i JavaScript} super-sets, adding optional {\i strong typing}, and classes, close to the {\i ECMAScript Sixth Edition}: the current main language in this category is certainly {\i TypeScript}, designed by Anders Hejlsberg (father of both the {\i Delphi} language and {\i C#}), and published by {\i Microsoft};
- New languages, dedicated to make writing {\i JavaScript} programs easier, with an alternative syntax and new concepts (like classes, lambdas, scoping, splats, comprehensions...): most relevant languages of this family are {\i CoffeeScript} and {\i Dart};
- High-level languages, like {\i Google Web Toolkit} (compiling {\i Java} code), {\i JSIL} (from {\i C#} via {\i Mono}), or {\i Smart Mobile Studio} (from {\i object pascal});
- Low-level languages, like {\i Emscripten} (compiling C/C++ from LLVM byte-code, using {\i asm.js}).
Of course, from our point of view, use of modern {\i object pascal} is of great interest, since it will leverage our own coding skills, and make us able to share code between client and server sides.
:   Beyond JavaScript
The {\i Smart Pascal} language brings strong typing, true @*OOP@ to {\i JavaScript}, including classes, partial classes, interfaces, inheritance, polymorphism, virtual and abstract classes and methods, helpers, closures, lambdas, enumerations and sets, getter/setter expressions, operator overloading, contract programming. But you can still unleash the power of {\i JavaScript} (some may say "the good parts"), if needed: the {\f1\fs20 variant} type is used to allow dynamic typing, and you can write some {\i JavaScript} code as an {\f1\fs20 asm .. end} block.\line See @http://en.wikipedia.org/wiki/The_Smart_Pascal_programming_language
The resulting HTML5 project is self-sufficient with no external {\i JavaScript} library, and is compiled as a single {\f1\fs20 index.html} file (including its {\f1\fs20 css}, if needed). The {\i JavaScript} code generated by the compiler (written in {\i Delphi} by Eric Grange), is of very high quality, optimized for best execution performance (either in JIT or V8), has low memory consumption, and can be compressed and/or obfuscated.
The {\f1\fs20 SmartCL} runtime library encapsulate HTML5 APIs in a set of pure pascal classes and functions, and an IDE with an integrated form designer is available. You can debug your application directly within the IDE (since revision 2.1 - even if it is not yet always stable) or within your browser (IE, Chrome or FireBug have great debuggers), with step-by-step execution of the object pascal code (if you define "{\i Add source map (for debugging)}" in {\f1\fs20 Project Options} / {\f1\fs20 Linker}).
Using a third-party tool like {\i @*PhoneGap@} - see @http://phonegap.com - you will be able to supply your customers with true native {\i iOS} or {\i Android} applications, running without any network, and accessing the full power of any modern {\i Smart Phone}. Resulting applications will be much smaller in size than the one generated by {\i Delphi} FMX (a simple {\i Smart} RESTful client with a login form and ORM + SOA tests is zipped as 40 KB), and will work seamlessly on all HTML5 platforms, including most mobile (like Windows Phone, Blackberry, Firefox OS, or webOS) or desktop (Windows, @*Linux@, BSD, MacOS) architectures.
{\i @*Smart Mobile Studio@} is therefore a great platform for implementing rich client-side AJAX or {\i Mobile} applications, to work with our client-server {\i mORMot} framework.
:   Using Smart Mobile Studio with mORMot
There is no package to be installed within the {\i Smart Mobile Studio} IDE. The client units will be generated directly from the {\i mORMot} server.\line Any edition of {\i Smart} - see @http://smartmobilestudio.com/feature-matrix - is enough: you do not need to pay for the {\i Enterprise} edition to consume {\i mORMot} services. But of course, the {\i Professionnal} edition is recommended, since the {\i Basic} edition does not allow to create forms from the IDE, which is the main point for an AJAX application.
In contrast to the wrappers available in the {\i Professional} edition of Smart, for accessing {\i RemObjects} or {\i DataSnap} servers, our {\i mORMot} clients are 100% written in the {\i @*SmartPascal@} dialect. There is no need to link an external {\f1\fs20 .js} library to your executable, and you will benefit of the obfuscation and smart linking features of the Smart compiler.
The only requirement is to copy the {\i mORMot} cross-platform units to your {\i Smart Mobile Studio} installation. This can be done in three {\f1\fs20 copy} instructions:
$xcopy SynCrossPlatformSpecific.pas "c:\ProgramData\Optimale Systemer AS\Smart Mobile Studio\Libraries" /Y
$xcopy SynCrossPlatformCrypto.pas "c:\ProgramData\Optimale Systemer AS\Smart Mobile Studio\Libraries" /Y
$xcopy SynCrossPlatformREST.pas "c:\ProgramData\Optimale Systemer AS\Smart Mobile Studio\Libraries" /Y
You can find a corresponding BATCH file in the {\f1\fs20 CrossPlatform} folder, and in {\f1\fs20 SQLite3\\Samples\\29 - SmartMobileStudio Client\\CopySynCrossPlatformUnits.bat}.
In fact, the {\f1\fs20 SynCrossPlatformJSON.pas} unit is not used under {\i Smart Mobile Studio}: we use the built-in JSON serialization features of {\i JavaScript}, using {\f1\fs20 variant} dynamic type, and the standard {\f1\fs20 JSON.Stringify()} and {\f1\fs20 JSON.Parse()} functions.
:  Remote logging
Since there is no true file system API available under a HTML5 sand-boxed application, logging to a local file is not an option. Even when packaged with {\i PhoneGap}, local log files are not convenient to work with.
Generated logs will have the same methods and format as with {\i Delphi} or {\i FreePascal} - see @105@. {\f1\fs20 TSQLRest.Log(E: Exception)} method will also log the stack trace of the exception!  Our {\f1\fs20 LogView} tool - see @103@ - is able to run as a simple but efficient remote log server and viewer, shared with regular or cross-platform units of the framework.
A dedicated {\i asynchronous} implementation has been refined for {\i Smart Mobile Studio} clients, so that several events will be gathered and sent at once to the remote server, to maximize bandwidth use and let the application be still responsive.\line It allows even complex mobile applications to be debugged with ease, on any device, even over WiFi or 3G/4G networks. Your support could ask your customer to enable logging for a particular case, then see in real time what is wrong with your application.
\page
: Generating client wrappers
Even if it is feasible to write the client code by hand, your {\i mORMot} server is able to create the source code needed for client access, via a dedicated method-based service, and a set of {\i @*Mustache@}-based templates - see @81@.
The following templates are available in the {\f1\fs20 CrossPlatform\\templates} folder:
|%45%45
|\b Unit Name|Compiler Target\b0
|{\f1\fs20 CrossPlatform.pas.mustache}|{\i Delphi} / FPC SynCrossPlatform* units
|{\f1\fs20 Delphi.pas.mustache}|{\i Delphi} Win32/Win64 {\i mORMot} units
|{\f1\fs20 SmartMobileStudio.pas.mustache}|@*Smart Mobile Studio@ 2.1
|%
In the future, other wrappers may be added. And you can write your own, which could be included within the framework source! Your input is warmly welcome, especially if you want to write a template for {\i Java} or {\f1\fs20 C#} client. The generated data context already contains the data types corresponding  to those compilers: e.g. a {\i mORMot}'s {\f1\fs20 RawUTF8} field or parameter could be identified as {\f1\fs20 "typeCS":"string"} or {\f1\fs20 "typeJava":"String"} in addition to {\f1\fs20 "typeDelphi":"RawUTF8"} and {\f1\fs20 "typePascal":"string"}.
:  Publishing the code generator
By default, and for security reasons, the code generation is not embedded to your {\i mORMot} RESTful server. In fact, the {\f1\fs20 mORMotWrapper.pas} unit will link both {\f1\fs20 mORMot.pas} and {\f1\fs20 SynMustache.pas} units, and use {\i Mustache} templates to generate code for a given {\f1\fs20 TSQLRestServer} instance.
We will start from the interface-based service @89@ as defined in the\line "{\f1\fs20 SQLite3\\Samples\\14 - Interface based services}" folder.\line After some minor modifications, we copied the server source code into\line "{\f1\fs20 SQLite3\\Samples\\27 - CrossPlatform Clients\\Project14ServerHttpWrapper.dpr}":
!program Project14ServerHttpWrapper;
!
!{$APPTYPE CONSOLE}
!
!uses
!  SysUtils,
!  Classes,
!  SynCommons,
!  mORMot,
!  mORMotHttpServer,
!!  mORMotWrappers,
!  Project14Interface in '..\14 - Interface based services\Project14Interface.pas';
!
!type
!  TServiceCalculator = class(TInterfacedObject, ICalculator)
!  public
!    function Add(n1,n2: integer): integer;
!  end;
!
!function TServiceCalculator.Add(n1, n2: integer): integer;
!begin
!  result := n1+n2;
!end;
!
!var
!  aModel: TSQLModel;
!  aServer: TSQLRestServer;
!  aHTTPServer: TSQLHttpServer;
!begin
!  // create a Data Model
!  aModel := TSQLModel.Create([],ROOT_NAME);
!  try
!    // initialize a TObjectList-based database engine
!    aServer := TSQLRestServerFullMemory.Create(aModel,'test.json',false,true);
!    try
!!      // add the http://localhost:888/root/wrapper code generation web page
!!      AddToServerWrapperMethod(aServer,
!!        ['..\..\..\CrossPlatform\templates','..\..\..\..\CrossPlatform\templates']);
!      // register our ICalculator service on the server side
!      aServer.ServiceDefine(TServiceCalculator,[ICalculator],sicShared);
!      // launch the HTTP server
!      aHTTPServer := TSQLHttpServer.Create(PORT_NAME,[aServer],'+',useHttpApiRegisteringURI);
!      try
!        aHTTPServer.AccessControlAllowOrigin := '*'; // for AJAX requests to work
!        writeln(#10'Background server is running.');
!        writeln('You can test http://localhost:',PORT_NAME,'/wrapper');
!        writeln(#10'Press [Enter] to close the server.'#10);
!        readln;
!      finally
!        aHTTPServer.Free;
!      end;
!    finally
!      aServer.Free;
!    end;
!  finally
!    aModel.Free;
!  end;
!end.
As you can see, we just added a reference to the {\f1\fs20 mORMotWrappers} unit, and a call to {\f1\fs20 AddToServerWrapperMethod()} in order to publish the available code generators.
Now, if you run the {\f1\fs20 Project14ServerHttpWrapper} server, and point your favorite browser to {\f1\fs20 http://localhost:888/root/wrapper} you will see the following page:\line
{\b\fs28 Client Wrappers}
{\b Available Templates:}
* {\b CrossPlatform}\line {\i mORMotClient.pas} - {\ul download as file} - {\ul see as text} - {\ul see template}
* {\b Delphi}\line {\i mORMotClient.pas} - {\ul download as file} - {\ul see as text} - {\ul see template}
* {\b SmartMobileStudio}\line {\i mORMotClient.pas} - {\ul download as file} - {\ul see as text} - {\ul see template}
You can also retrieve the corresponding {\ul template context}.\line
Each of the {\f1\fs20 *.mustache} template available in the specified folder is listed here. Links above will allow downloading a client source code unit, or displaying it as text in the browser. The template can also be displayed un-rendered, for reference. As true {\i Mustache} templates, the source code files are generated from a {\i data context}, which can be displayed, as JSON, from the "{\ul template context}" link. It may help you when debugging your own templates. Note that if you modify and save a {\f1\fs20 .mustache} template file, just re-load the "{\ul see as text}" browser page and your modification is taken in account immediately (you do not need to restart the server).
Generated source code will follow the template name, and here will always be downloaded as {\f1\fs20 mORMotClient.pas}. Of course, you can change the unit name for your end-user application. It could be even mandatory if the same client will access to several {\i mORMot} servers at once, which could be the case in a @17@ project.
Just ensure that you will never change the {\f1\fs20 mORMotClient.pas} generated content by hand. If necessary, you can create and customize your own {\i Mustache} template, to be used for your exact purpose. By design, such automated code generation will require to re-create the client unit each time the server ORM or SOA structure is modified. In fact, as stated in the {\f1\fs20 mORMotClient.pas} comment, any manual modification of this file may be lost after regeneration. You have been warned!
For publishing the wrappers for a REST / ORM oriented program, take a look at the '{\f1\fs20 28 - Simple RESTful ORM Server}' sample.
If you feel that the current templates have some issues or need some enhancements, you are very welcome to send us your change requests on our forums. Once you are used at it, {\i Mustache} templates are fairly easy to work with. Similarly, if you find out that some information is missing in the generated {\i data context}, e.g. for a new platform or language, we will be pleased to enhance the official {\f1\fs20 mORMotWrapper.pas} process.
:  Delphi / FreePascal client samples
The "{\f1\fs20 27 - CrossPlatform Clients\RegressionTests.dpr}" sample creates a {\i mORMot} server with its own ORM data model, containing a {\f1\fs20 TSQLRecordPeople} class, and a set of interface-based SOA services, some including complex types like a record.
Then this sample uses a generated {\f1\fs20 mORMotClient.pas}, retrieved from the "{\ul download as file}" link of the {\b CrossPlatform} template above.\line Its set of regression tests (written using a small cross-platform {\f1\fs20 TSynTest} unit test class) will then perform remote ORM and SOA calls to the {\f1\fs20 PeopleServer} embedded instance, over all supported authentication schemes - see @18@:
$ Cross Platform Units for mORMot
$---------------------------------
$
$ 1. Running "Iso8601DateTime"
$    30003 tests passed in 00:00:018
$ 2. Running "Base64Encoding"
$    304 tests passed in 00:00:000
$ 3. Running "JSON"
$    18628 tests passed in 00:00:056
$ 4. Running "Model"
$    1013 tests passed in 00:00:003
$ 5. Running "Cryptography"
$    4 tests passed in 00:00:000
$
$ Tests failed: 0 / 49952
$ Time elapsed: 00:00:080
$
$ Cross Platform Client for mORMot without authentication
$---------------------------------------------------------
$
$ 1. Running "Connection"
$    2 tests passed in 00:00:010
$ 2. Running "ORM"
$    4549 tests passed in 00:00:160
$ 3. Running "ORMBatch"
$    4564 tests passed in 00:00:097
$ 4. Running "Services"
$    26253 tests passed in 00:00:302
$ 5. Running "CleanUp"
$    1 tests passed in 00:00:000
$
$ Tests failed: 0 / 35369
$ Time elapsed: 00:00:574
$
$ Cross Platform Client for mORMot using TSQLRestServerAuthenticationNone
$-------------------------------------------------------------------------
$  ...
$
$ Cross Platform Client for mORMot using TSQLRestServerAuthenticationDefault
$----------------------------------------------------------------------------
$  ...
The generated {\f1\fs20 mORMotClient.pas} unit is used for all "{\f1\fs20 Cross Platform Client}" tests above, covering both ORM and SOA features of the framework.
:   Connection to the server
You could manually connect to a {\i mORMot} server as such:
!var Model: TSQLModel;
!    Client: TSQLRestClientHTTP;
!...
!  Model := TSQLModel.Create([TSQLAuthUser,TSQLAuthGroup,TSQLRecordPeople]);
!  Client := TSQLRestClientHTTP.Create('localhost',SERVER_PORT,Model);
!  if not Client.Connect then
!    raise Exception.Create('Impossible to connect to the server');
!  if Client.ServerTimestamp=0 then
!    raise Exception.Create('Incorrect server');
!  if not Client.SetUser(TSQLRestAuthenticationDefault,'User','synopse') then
!    raise Exception.Create('Impossible to authenticate to the server');
!...
Or you may use the {\f1\fs20 GetClient()} function generated in {\f1\fs20 mORMotClient.pas}:
!/// create a TSQLRestClientHTTP instance and connect to the server
!// - it will use by default port 888
!// - secure connection will be established via TSQLRestServerAuthenticationDefault
!// with the supplied credentials - on connection or authentication error,
!// this function will raise a corresponding exception
!function GetClient(const aServerAddress, aUserName,aPassword: string;
!  aServerPort: integer=SERVER_PORT): TSQLRestClientHTTP;
Which could be used as such:
!var Client: TSQLRestClientHTTP;
!...
!  Client := GetClient('localhost','User','synopse')
The data model and the expected authentication scheme were included in the {\f1\fs20 GetClient()} function, which will raise the expected {\f1\fs20 ERestException} in case of any connection or authentication issue.
:   CRUD/ORM remote access
Thanks to {\f1\fs20 SynCrossPlatform*} units, you could easily perform any remote ORM operation on your {\i mORMot} server, with the usual {\f1\fs20 TSQLRest} CRUD methods.\line For instance, the {\f1\fs20 RegressionTests.dpr} sample performs the following operations
!!  fClient.CallBackGet('DropTable',[],Call,TSQLRecordPeople); // call of method-based service
!  check(Call.OutStatus=HTTP_SUCCESS);
!  people := TSQLRecordPeople.Create; // create a record ORM
!  try
!    for i := 1 to 200 do begin
!      people.FirstName := 'First'+IntToStr(i);
!      people.LastName := 'Last'+IntToStr(i);
!      people.YearOfBirth := i+1800;
!      people.YearOfDeath := i+1825;
!      people.Sexe := TPeopleSexe(i and 1);
!!      check(Client.Add(people,true)=i); // add one record
!    end;
!  finally
!    people.Free;
!  end;
!...
!!  people := TSQLRecordPeople.CreateAndFillPrepare(fClient,'',
!!    'yearofbirth=?',[1900]); // parameterized query returning one or several rows
!  try
!    n := 0;
!!    while people.FillOne do begin
!      inc(n);
!      check(people.ID=100);
!      check(people.FirstName='First100');
!      check(people.LastName='Last100');
!      check(people.YearOfBirth=1900);
!      check(people.YearOfDeath=1925);
!    end;
!    check(n=1); // we expected only one record here
!  finally
!    people.Free;
!  end;
!  for i := 1 to 200 do
!    if i and 15=0 then
!!      fClient.Delete(TSQLRecordPeople,i) else // record deletion
!    if i mod 82=0 then begin
!      people := TSQLRecordPeople.Create;
!      try
!        id := i+1;
!        people.ID := i;
!        people.YearOfBirth := id+1800;
!        people.YearOfDeath := id+1825;
!!        check(fClient.Update(people,'YEarOFBIRTH,YEarOfDeath')); // record modification
!      finally
!        people.Free;
!      end;
!    end;
!  for i := 1 to 200 do begin
!!    people := TSQLRecordPeople.Create(fClient,i); // retrieve one instance from ID
!    try
!      if i and 15=0 then // was deleted
!        Check(people.ID=0) else begin
!        if i mod 82=0 then
!          id := i+1 else // was modified
!          id := i;
!        Check(people.ID=i);
!        Check(people.FirstName='First'+IntToStr(i));
!        Check(people.LastName='Last'+IntToStr(i));
!        Check(people.YearOfBirth=id+1800);
!        Check(people.YearOfDeath=id+1825);
!        Check(ord(people.Sexe)=i and 1);
!      end;
!    finally
!      people.Free;
!    end;
!  end;
As we already stated, @*BATCH@ mode is also supported, with the classic {\i mORMot} syntax:
!...
!    res: TIntegerDynArray;
!...
!!  fClient.BatchStart(TSQLRecordPeople);
!  people := TSQLRecordPeople.Create;
!  try
!    for i := 1 to 200 do begin
!      people.FirstName := 'First'+IntToStr(i);
!      people.LastName := 'Last'+IntToStr(i);
!      people.YearOfBirth := i+1800;
!      people.YearOfDeath := i+1825;
!!      fClient.BatchAdd(people,true);
!    end;
!  finally
!    people.Free;
!  end;
!!  fClient.fBatchSend(res)=HTTP_SUCCESS);
!  check(length(res)=200);
!  for i := 1 to 200 do
!    check(res[i-1]=i); // server returned the IDs of the newly created records
Those {\f1\fs20 BatchAdd} / {\f1\fs20 BatchDelete} / {\f1\fs20 BatchUpdate} methods of {\f1\fs20 TSQLRest} have the benefit to introduce at client level:
- Much higher performance, especially on multi-insertion or multi-update of data;
- Transactional support: {\f1\fs20 TSQLRest.BatchStart()} has an optional {\f1\fs20 @*AutomaticTransactionPerRow@} parameter, set to {\f1\fs20 10000} by default, which will create a server-side transaction during the write process, enable @78@ or @99@ on the server side if available, and an ACID rollback in case of any failure.
You can note that all above code has exactly the same structure and methods than standard {\i mORMot} clients.
The generated {\f1\fs20 mORMotClient.pas} unit contains all needed {\f1\fs20 TSQLRecord} types, and its used properties, including enumerations or complex records. The only dependency of this unit are {\f1\fs20 SynCrossPlatform*} units, so will be perfectly cross-platform (whereas our main {\f1\fs20 SynCommons.pas} and {\f1\fs20 mORMot.pas} units do target only {\i Win32} and {\i Win64}).
As a result, you are able to {\i share} server and client code between a Windows project and any supported platform, even AJAX (see "{\i Smart Mobile Studio client samples}" below). A shared unique code base will eventually reduce both implementation and debugging time, which is essential to unleash your business code potential and maximize your ROI.
:   Service consumption
The ultimate goal of the {\i mORMot} framework is to publish your business via a @17@.\line As a consequence, those services should be made available from any kind of device or platform, even outside the {\i Windows} world. The server is able to generate client wrappers code, which could be used to consume any @63@ using any supported authentication scheme - see @18@.
Here is an extract of the {\f1\fs20 mORMotClient.pas} unit as generated for the {\f1\fs20 RegressionTests.dpr} sample:
!type
!  /// service implemented by TServiceCalculator
!  // - you can access this service as such:
!  // !var aCalculator: ICalculator;
!  // !begin
!  // !   aCalculator := TCalculator.Create(aClient);
!  // !   // now you can use aCalculator methods
!  // !...
!  ICalculator = interface(IServiceAbstract)
!    ['{9A60C8ED-CEB2-4E09-87D4-4A16F496E5FE}']
!    function Add(const n1: integer; const n2: integer): integer;
!    procedure ToText(const Value: currency; const Curr: string; var Sexe: TPeopleSexe; var Name: string);
!    function RecordToText(var Rec: TTestCustomJSONArraySimpleArray): string;
!  end;
!
!  /// implements ICalculator from http://localhost:888/root/Calculator
!  // - this service will run in sicShared mode
!  TServiceCalculator = class(TServiceClientAbstract,ICalculator)
!  public
!    constructor Create(aClient: TSQLRestClientURI); override;
!    function Add(const n1: integer; const n2: integer): integer;
!    procedure ToText(const Value: currency; const Curr: string; var Sexe: TPeopleSexe; var Name: string);
!    function RecordToText(var Rec: TTestCustomJSONArraySimpleArray): string;
!  end;
As you can see, a dedicated class has been generated to consume the server-side {\f1\fs20 ICalculator} interface-based service, in its own {\f1\fs20 ICalculator} client-side type.\line It is able to handle complex types, like enumerations (e.g. {\f1\fs20 TPeopleSexe}) and records (e.g. {\f1\fs20 TTestCustomJSONArraySimpleArray}), which are also defined in the very same {\f1\fs20 mORMotClient.pas} unit.\line You can note that the {\f1\fs20 RawUTF8} type has been changed into the standard {\i Delphi} / {\i @*FreePascal@ }{\f1\fs20 string} type, since it is the native type used by our {\f1\fs20 SynCrossPlatformJSON.pas} unit for all its JSON marshalling. Of course, under latest version of {\i Delphi} and {\i FreePascal}, this kind of content may be Unicode encoded (either as UTF-16 for the {\f1\fs20 string} = {\f1\fs20 UnicodeString} {\i Delphi} type, or as @*UTF-8@ for the {\i FreePascal} / {\i @*Lazarus@} {\f1\fs20 string} type).
The supplied regression tests show how to use remotely those services:
!!var calc: ICalculator;
!    i,j: integer;
!    sex: TPeopleSexe;
!    name: string;
!...
!!  calc := TServiceCalculator.Create(fClient);
!  check(calc.InstanceImplementation=sicShared);
!  check(calc.ServiceName='Calculator');
!  for i := 1 to 200 do
!!    check(calc.Add(i,i+1)=i*2+1);
!  for i := 1 to 200 do begin
!    sex := TPeopleSexe(i and 1);
!    name := 'Smith';
!!    calc.ToText(i,'$',sex,name);
!    check(sex=sFemale);
!    check(name=format('$ %d for %s Smith',[i,SEX_TEXT[i and 1]]));
!  end;
!...
As with regular {\i mORMot} client code, a {\f1\fs20 TServiceCalculator} instance is created and is assigned to a {\f1\fs20 ICalculator} local variable. As such, no {\f1\fs20 try ... finally Calc.Free end} block is mandatory here, to avoid any memory leak: the compiler will create such an hidden block for the {\f1\fs20 Calc: ICalculator} variable scope.
The service-side contract of the {\f1\fs20 ICalculator} signature is retrieved and checked within {\f1\fs20 TServiceCalculator.Create}, and will raise an {\f1\fs20 ERestException} if it does not match the contract identified in {\f1\fs20 mORMotClient.pas}.
The cross-platform clients are able to manage the service instance life-time, especially the {\f1\fs20 sicPerClient} mode. In this case, an implementation class instance will be created on the server for each client, until the corresponding {\f1\fs20 interface} instance will released (i.e. out of scope or assigned to {\f1\fs20 nil}), which will release the server-side instance - just like with a regular {\i mORMot} client code.
Note that all process here is executed {\i synchronously}, i.e. in blocking mode. It is up to you to ensure that your application is able to still be responsive, even if the server does a lot of process, so may be late to answer. A dedicated thread may help in this case.
:116  Smart Mobile Studio client samples
In addition to {\i Delphi} and {\i FreePascal} clients, our framework is able to access any {\i mORMot} server from HTML5 / AJAX rich client, thanks to {\i @*Smart Mobile Studio@}.
:   Adding two numbers in AJAX
You can find in {\f1\fs20 SQLite3\\Samples\\27 - CrossPlatform Clients\\SmartMobileStudio} a simple client for the {\f1\fs20 TServiceCalculator.Add()} interface-based service.\line If your {\f1\fs20 Project14ServerHttpWrapper} server is running, you can just point to the supplied {\f1\fs20 www\\index.html} file in the sub-folder.\line You will then see a web page with a "{\f1\fs20 Server Connect}" button, and if you click on it, you will be able to add two numbers. This a full HTML5 @*web application@, connecting securely to your {\i mORMot} server, which will work from any desktop browser (on {\i Windows}, {\i Mac OS X}, or {\i @*Linux@}), or from any mobile device (either {\i @*iPhone@} / {\i @*iPad@} / {\i @*Android@} / {\i Windows 8 Mobile}).
In order to create the application, we just clicked on "{\ul download as file}" in the {\b SmartMobileStudio} link in the web page, and copied the generated file in the source folder of a new {\i Smart Mobile} project.\line Of course, we did copy the needed {\f1\fs20 SynCrossPlatform*.pas} units from the {\i mORMot} source code tree into the Smart library folder, as stated above. Just ensure you run {\f1\fs20 CopySynCrossPlatformUnits.bat} from the {\f1\fs20 CrossPlatform} folder at least once from the latest revision of the framework source code.
Then, on the form visual editor, we added a {\f1\fs20 BtnConnect} button, then a {\f1\fs20 PanelCompute} panel with two edit fields named {\f1\fs20 EditA} and {\f1\fs20 EditB}, and two other buttons, named {\f1\fs20 BtnComputeAsynch} and {\f1\fs20 BtnComputeSynch}. A {\f1\fs20 LabelResult} label will be used to display the computation result. The {\f1\fs20 BtnConnect} is a toggle which will show or display the {\f1\fs20 PanelCompute} panel, which is hidden by default, depending on the connection status.
%SmartCalculator.png
In the {\f1\fs20 Form1.pas}  unit source code side, we added a reference to our both {\f1\fs20 SynCrossPlatformREST} and {\f1\fs20 mORMotClient} units, and some events to the buttons:
!unit Form1;
!
!interface
!
!uses
!  SmartCL.System, SmartCL.Graphics, SmartCL.Components, SmartCL.Forms,
!  SmartCL.Fonts, SmartCL.Borders, SmartCL.Application, SmartCL.Controls.Panel,
!  SmartCL.Controls.Label, SmartCL.Controls.EditBox, SmartCL.Controls.Button,
!!  SynCrossPlatformREST, mORMotClient;
!
!type
!  TForm1 = class(TW3Form)
!    procedure BtnComputeSynchClick(Sender: TObject);
!    procedure BtnComputeAsynchClick(Sender: TObject);
!    procedure BtnConnectClick(Sender: TObject);
!  private
!    {$I 'Form1:intf'}
!  protected
!!    Client: TSQLRestClientURI;
!    procedure InitializeForm; override;
!    procedure InitializeObject; override;
!    procedure Resize; override;
!  end;
The {\f1\fs20 BtnConnect} event will connect asynchronously to the server, using {\f1\fs20 'User'} as log-on name, and {\f1\fs20 'synopse'} as password (those as the framework defaults).\line We just use the {\f1\fs20 GetClient()} function, as published in our generated {\f1\fs20 mORMotClient.pas} unit:
!/// create a TSQLRestClientHTTP instance and connect to the server
!// - it will use by default port 888
!// - secure connection will be established via TSQLRestServerAuthenticationDefault
!// with the supplied credentials
!// - request will be asynchronous, and trigger onSuccess or onError event
!procedure GetClient(const aServerAddress, aUserName,aPassword: string;
!  onSuccess, onError: TSQLRestEvent; aServerPort: integer=SERVER_PORT);
It uses two callbacks, the first in case of success, and the second triggered on failure. On success, we will set the global {\f1\fs20 Client} variable with the {\f1\fs20 TSQLRestClientURI} instance just created, then display the two fields and compute buttons:
!procedure TForm1.BtnConnectClick(Sender: TObject);
!begin
!  if Client=nil then
!!    GetClient('127.0.0.1','User','synopse',
!!      lambda (aClient: TSQLRestClientURI)
!        PanelCompute.Visible := true;
!        W3Label1.Visible := true;
!        W3Label2.Visible := true;
!        LabelConnect.Caption := '';
!        BtnConnect.Caption := 'Disconnect';
!        LabelResult.Caption := '';
!!        Client := aClient;
!      end,
!      lambda
!        ShowMessage('Impossible to connect to the server!');
!      end)
!  else begin
!    PanelCompute.Visible := false;
!    BtnConnect.Caption := 'Server Connect';
!    Client.Free;
!    Client := nil;
!  end;
!end;
The {\f1\fs20 GetClient()} function expects two callbacks, respectively {\f1\fs20 onSuccess} and {\f1\fs20 onError}, which are implemented here with two {\i SmartPascal} {\f1\fs20 lambda} blocks.
Now that we are connected to the server, let's do some useful computation!\line As you can see in the {\f1\fs20 mORMotClient.pas} generated unit, our interface-based service can be accessed via a {\i @*SmartPascal@} {\f1\fs20 TServiceCalculator} class (and not an {\f1\fs20 interface}), with two variations of each methods: one {\i asynchronous} method - e.g. {\f1\fs20 TServiceCalculator.Add()} - expecting success/error callbacks, and one {\i synchronous} (blocking) method - e.g. {\f1\fs20 TServiceCalculator._Add()}:
!type
!  /// service accessible via http://localhost:888/root/Calculator
!  // - this service will run in sicShared mode
!  // - synchronous and asynchronous methods are available, depending on use case
!  // - synchronous _*() methods will block the browser execution, so won't be
!  // appropriate for long process - on error, they may raise EServiceException
!  TServiceCalculator = class(TServiceClientAbstract)
!  public
!    /// will initialize an access to the remote service
!    constructor Create(aClient: TSQLRestClientURI); override;
!    procedure Add(n1: integer; n2: integer;
!      onSuccess: procedure(Result: integer); onError: TSQLRestEvent);
!    function _Add(const n1: integer; const n2: integer): integer;
!  end;
We can therefore execute asynchronously the {\f1\fs20 Add()} service as such:
!procedure TForm1.BtnComputeAsynchClick(Sender: TObject);
!begin
!  TServiceCalculator.Create(Client).Add(
!    StrToInt(EditA.Text),StrToInt(EditB.Text),
!    lambda (res: integer)
!      LabelResult.Caption := format('Result = %d',[res]);
!    end,
!    lambda
!      ShowMessage('Error calling the method!');
!    end);
!end;
Or execute synchronously the {\f1\fs20 _Add()} service:
!procedure TForm1.BtnComputeSynchClick(Sender: TObject);
!begin
!  LabelResult.Caption := format('Result = %d',
!    [TServiceCalculator.Create(Client)._Add(
!      StrToInt(EditA.Text),StrToInt(EditB.Text))]);
!end;
Of course, the synchronous code is much easier to follow and maintain. To be fair, the {\i SmartPascal} {\f1\fs20 lambda} syntax is not difficult to read nor write. In the browser debugger, you can easily set a break point within any {\f1\fs20 lambda} block, and debug your code.
Note that if the server is slow to answer, your whole web application will be unresponsive, and the browser may even complain about the page, proposing the kill its process!\line As a consequence, simple services may be written in a synchronous manner, but your serious business code should rather use asynchronous callbacks, just as with any modern AJAX application.
Thanks to the {\i Smart Linking} feature of its compiler, only the used version of the unit will be converted to {\i JavaScript} and included in the final {\f1\fs20 index.html} HTML5 file. So having both synchronous and asynchronous versions of each method at hand is not an issue.
:   CRUD/ORM remote access
If the server did have some ORM model, its {\f1\fs20 TSQLRecord} classes will also be part of the {\f1\fs20 mORMotClient.pas} generated unit. All types, even complex record structures, will be marshaled as expected.
For instance, if you run the {\f1\fs20 RegressionTestsServer.dpr} server (available in the same folder), a much more complete unit could be generated from {\f1\fs20 http://localhost:888/root/wrapper}:
!type // define some enumeration types, used below
!  TPeopleSexe = (sFemale, sMale);
!  TRecordEnum = (reOne, reTwo, reLast);
!
!type // define some record types, used as properties below
!  TTestCustomJSONArraySimpleArray = record
!    F: string;
!    G: array of string;
!    H: record
!      H1: integer;
!      H2: string;
!      H3: record
!        H3a: boolean;
!        H3b: TSQLRawBlob;
!      end;
!    end;
!    I: TDateTime;
!    J: array of record
!      J1: byte;
!      J2: TGUID;
!      J3: TRecordEnum;
!    end;
!  end;
!
!type
!  /// service accessible via http://localhost:888/root/Calculator
!  // - this service will run in sicShared mode
!  // - synchronous and asynchronous methods are available, depending on use case
!  // - synchronous _*() methods will block the browser execution, so won't be
!  // appropriate for long process - on error, they may raise EServiceException
!  TServiceCalculator = class(TServiceClientAbstract)
!  public
!    /// will initialize an access to the remote service
!    constructor Create(aClient: TSQLRestClientURI); override;
!    procedure Add(n1: integer; n2: integer;
!      onSuccess: procedure(Result: integer); onError: TSQLRestEvent);
!    function _Add(const n1: integer; const n2: integer): integer;
!    procedure ToText(Value: currency; Curr: string; Sexe: TPeopleSexe; Name: string;
!      onSuccess: procedure(Sexe: TPeopleSexe; Name: string); onError: TSQLRestEvent);
!    procedure _ToText(const Value: currency; const Curr: RawUTF8; var Sexe: TPeopleSexe; var Name: RawUTF8);
!    procedure RecordToText(Rec: TTestCustomJSONArraySimpleArray;
!      onSuccess: procedure(Rec: TTestCustomJSONArraySimpleArray; Result: string); onError: TSQLRestEvent);
!    function _RecordToText(var Rec: TTestCustomJSONArraySimpleArray): string;
!  end;
!
!  /// map "People" table
!  TSQLRecordPeople = class(TSQLRecord)
!  protected
!    fFirstName: string;
!    fLastName: string;
!    fData: TSQLRawBlob;
!    fYearOfBirth: integer;
!    fYearOfDeath: word;
!    fSexe: TPeopleSexe;
!    fSimple: TTestCustomJSONArraySimpleArray;
!    // those overriden methods will emulate the needed RTTI
!    class function ComputeRTTI: TRTTIPropInfos; override;
!    procedure SetProperty(FieldIndex: integer; const Value: variant); override;
!    function GetProperty(FieldIndex: integer): variant; override;
!  public
!    property FirstName: string read fFirstName write fFirstName;
!    property LastName: string read fLastName write fLastName;
!    property Data: TSQLRawBlob read fData write fData;
!    property YearOfBirth: integer read fYearOfBirth write fYearOfBirth;
!    property YearOfDeath: word read fYearOfDeath write fYearOfDeath;
!    property Sexe: TPeopleSexe read fSexe write fSexe;
!    property Simple: TTestCustomJSONArraySimpleArray read fSimple write fSimple;
!  end;
In the above code, you can see several methods to the {\f1\fs20 ICalculator} service, some involving the complex {\f1\fs20 TTestCustomJSONArraySimpleArray} record type. The {\f1\fs20 implementation} section of the unit will in fact allow serialization of such records to/from JSON, even with obfuscated {\i JavaScript} field names.
Some {\i enumerations} types are also defined, so will help your business code be very expressive, thanks to the {\i @*SmartPascal@} strong typing. This is a huge improvement when compared to {\i JavaScript} native weak and dynamic typing.
There is a {\f1\fs20 TSQLRecordPeople} class generated, which will map the following {\i Delphi} class type, as defined in the {\f1\fs20 PeopleServer.pas} unit:
!  TSQLRecordPeople = class(TSQLRecord)
!  protected
!    fData: TSQLRawBlob;
!    fFirstName: RawUTF8;
!    fLastName: RawUTF8;
!    fYearOfBirth: integer;
!    fYearOfDeath: word;
!    fSexe: TPeopleSexe;
!    fSimple: TTestCustomJSONArraySimpleArray;
!  public
!    class procedure InternalRegisterCustomProperties(Props: TSQLRecordProperties); override;
!  published
!    property FirstName: RawUTF8 read fFirstName write fFirstName;
!    property LastName: RawUTF8 read fLastName write fLastName;
!    property Data: TSQLRawBlob read fData write fData;
!    property YearOfBirth: integer read fYearOfBirth write fYearOfBirth;
!    property YearOfDeath: word read fYearOfDeath write fYearOfDeath;
!    property Sexe: TPeopleSexe read fSexe write fSexe;
!  public
!    property Simple: TTestCustomJSONArraySimpleArray read fSimple;
!  end;
Here, a complex {\f1\fs20 TTestCustomJSONArraySimpleArray} record field has been published, thanks to a manual {\f1\fs20 InternalRegisterCustomProperties()} registration, as we already stated above. Since {\i SmartPascal} is limited in terms of RTTI, the code generator did define some {\f1\fs20 ComputeRTTI() GetProperty()} and {\f1\fs20 SetProperty()} protected methods, which will, at runtime, perform all the properties marshalling to and from JSON.\line You can see that types like {\f1\fs20 RawUTF8} in the original {\i Delphi} {\f1\fs20 TSQLRecord} were mapped to the standard {\i SmartPascal} {\f1\fs20 string} type, as expected, when converted to the {\f1\fs20 mORMotClient.pas} generated unit.
Your AJAX client can then access to this {\f1\fs20 TSQLRecordPeople} content easily, via standard CRUD operations.\line See the {\f1\fs20 SQLite3\\Samples\\29 - SmartMobileStudio Client} sample, for instance the following line:
!!  people := new TSQLRecordPeople;
!  for i := 1 to 200 do begin
!!    assert(client.Retrieve(i,people));
!    assert(people.ID=i);
!    assert(people.FirstName='First'+IntToStr(i));
!    assert(people.LastName='Last'+IntToStr(i));
!    assert(people.YearOfBirth=id+1800);
!    assert(people.YearOfDeath=id+1825);
!  end;
Here, the {\f1\fs20 client} variable is a {\f1\fs20 TSQLRestClientURI} instance, as returned by the {\f1\fs20 GetClient() onSuccess} callback generated in {\f1\fs20 mORMotClient.pas}.\line You have {\f1\fs20 Add() Delete() Update() FillPrepare() CreateAndFillPrepare()} and {\f1\fs20 Batch*()} methods available, ready to safely access your data from your AJAX client.
If you update your data model on the server, just re-generate your {\f1\fs20 mORMotClient.pas} unit from {\f1\fs20 http://localhost:888/root/wrapper}, then rebuild your {\i Smart Mobile Studio} project to reflect all changes made to your ORM data model, or your SOA available services.
Thanks to the {\i SmartPascal} strong typing, any breaking change of the server expectations will immediately be reported at compilation, and not at runtime, as it will with regular {\i JavaScript} clients.
:MVC pattern
%cartoon08.png
The {\i mORMot} framwork allows writing rich and/or web MVC applications, relying on regular ORM and SOA methods to implement its business model and its application layer, with an optional dedicated MVC model for the HTML rendering.
: Model
According to the {\i @*Model@-View-Controller} (@*MVC@) pattern - see @10@ - the database schema should be handled separately from the User Interface.
The {\f1\fs20 @**TSQLModel@} class centralizes all {\f1\fs20 @*TSQLRecord@} inherited classes used by an application, both database-related and business-logic related.
See @110@ for how to define the model of your application.
: Views
The {\i mORMot} framework also features two kinds of {\i User Interface} generation, corresponding to the @*MVC@ {\i Views}:
- For Desktop clients written in {\i Delphi}, it allows creation of Ribbon-like interfaces, with full data view and navigation as visual Grids. Reporting and edition windows can be generated in an automated way. The whole User Interface is designed in code, by some constant definitions.
- For Web clients, an optimized @*Mustache@ @*Template@ engine in pure {\i Delphi} has been integrated, and allows easy creation of HTML views, with a clear MVC design.
\graph MVCWebClients MVC Web and Rich Clients
subgraph cluster_0 {
label="Web client 1";
"HTML Browser";
}
subgraph cluster_3 {
label="Web client 2";
"HTML Browser ";
}
subgraph cluster_4 {
label="Rich Client 3";
"FMX\nPresentation Tier";
}
subgraph cluster_1 {
label=" Server";
\HTML Browser \Web¤Presentation Tier
\HTML Browser\Web¤Presentation Tier
\Web¤Presentation Tier\Application Tier
\Application Tier\Business Logic Tier
}
\FMX¤Presentation Tier\Application Tier
subgraph cluster_2 {
label="  DB     Server";
\Business Logic Tier\Data Tier
}
\
The {\i Web Presentation Tier} will be detailed @108@, but we will now present the project-wide implementation proposal.
:  Desktop clients
:5   RTTI
The {\i Delphi} language (aka Object Pascal) provided Runtime Type Information (@**RTTI@) more than a decade ago. In short, Runtime Type Information is information about an object's data type that is set into memory at run-time. The RTTI support in {\i Delphi} has been added first and foremost to allow the design-time environment to do its job, but developers can also take advantage of it to achieve certain code simplifications. Our framework makes huge use of RTTI, from the database level to the User Interface. Therefore, the resulting program has the advantages of very fast development (Rails-like), but with the robustness of @*strong type@ syntax, and the speed of one of the best compiler available.
In short, it allows the software logic to be extracted from the code itself. Here are the places where this technology was used:
- All database structures are set in the code by normal classes definition, and most of the needed @*SQL@ code is created on the fly by the framework, before calling the {\i @*SQLite3@} database engine, resulting in a true Object-relational mapping (@*ORM@) framework;
- All User Interface is generated by the code, by using some simple data structures, relying on enumerations (see next paragraph);
- Most of the text displayed on the screen does rely on RTTI, thanks to the @*Camel@ approach (see below), ready to be translated into local languages;
- All internal Event process (such as Button press) relies on enumerations RTTI;
- Options and program parameters are using RTTI for data persistence and screen display (e.g. the Settings window of your program can be created by pure code): adding an option is a matter of a few code lines.
In {\i Delphi}, enumeration types or {\i Enum} provides a way of to define a list of values. The values have no inherent meaning, and their ordinality follows the sequence in which the identifiers are listed. These values are written once in the code, then used everywhere in the program, even for User Interface generation.
For example, some tool-bar actions can be defined with:
!type
!  /// item toolbar actions
!  TBabyAction = (
!    paCreateNew, paDelete, paEdit, paQuit);
Then this {\f1\fs20 TBabyAction} @*enumerated@ type is used to create the User Interface ribbon of the main window, just by creating an array of set of this kind:
!BarEdit: array[0..1] of set of TBabyAction = (
!    [paCreateNew, paDelete, paEdit],
!    [paQuit] );
The caption of the buttons to be displayed on the screen is then extracted by the framework using "@**Camel@ Case": the second button, defined by the {\f1\fs20 paCreateNew} identifier in the source code, is displayed as "{\i Create new}" on the screen, and this "{\i Create new}" is used for direct @*i18n@ of the software. For further information about "Camel Case" and its usage in Object Pascal, Java, Dot Net, Python see @http://en.wikipedia.org/wiki/CamelCase
Advantages of the RTTI can therefore by sum up:
- Software maintainability, since the whole program logic is code-based, and the User Interface is created from it. It therefore avoid RAD (Rapid Application Development) abuse, which mix the User Interface with data logic, and could lead into "write fast, try to maintain" scenarios;
- Enhanced code @*security@, thanks to Object Pascal @*strong type@ syntax;
- Direct database access from the language object model, without the need of writing @*SQL@ or use of a @*MVC@ framework;
- User Interface coherency, since most screen are created on the fly;
- Easy @*i18n@ of the software, without additional components or systems.
:64   User Interface
User Interface generation from RTTI and the integrated reporting features will be described @31@, during presentation of the Main Demo application design.
In short, such complex model including User Interface auto-creation could be written as such - extracted from unit {\f1\fs20 FileTables.pas}:
!function CreateFileModel(Owner: TSQLRest): TSQLModel;
!begin
!  result := TSQLModel.Create(Owner,
!    @FileTabs,length(FileTabs),sizeof(FileTabs[0]),[],
!    TypeInfo(TFileAction),TypeInfo(TFileEvent));
!end;
All needed {\f1\fs20 TSQLRecord} classes are declared in a
! FileTabs: array[0..4] of TFileRibbonTabParameters = ( ...
constant array, and will use {\f1\fs20 TFileAction / TFileEvent} enumeration types to handle the User Interface activity and Business Logic.
\page
:  Web clients
:81   Mustache template engine
{\i @**Mustache@} - see @http://mustache.github.io - is a well-known {\i logic-less} template engine.\line There is plenty of Open Source implementations around (including in {\i @*JavaScript@}, which can be very convenient for AJAX applications on client side, for instance). For {\i mORMot}, we created the first pure {\i Delphi} implementation of it, with a perfect integration with other bricks of the framework.
Generally speaking, a @**Template@ system can be used to separate output formatting specifications, which govern the appearance and location of output text and data elements, from the executable logic which prepares the data and makes decisions about what appears in the output.
Most template systems (e.g. PHP, smarty, Razor...) feature in fact a full scripting engine within the template content. It allows powerful constructs like variable assignment or conditional statements in the middle of the HTML content. It makes it easy to modify the look of an application within the template system exclusively, without having to modify any of the underlying "application logic". They do so, however, at the cost of separation, turning the templates themselves into part of the application logic.
{\i Mustache} inherits from Google's {\i ctemplate} library, and is used in many famous applications, including the "main" Google web search, or the Twitter web site.\line The {\i Mustache} template system leans strongly towards preserving the separation of logic and presentation, therefore ensures a perfect  @*MVC@ - @10@ - design, and ready to consume @*SOA@ services.
{\i Mustache} is intentionally constrained in the features it supports and, as a result, applications tend to require quite a bit of code to instantiate a template: all the application logic will be defined within the {\i Controller} code, not in the {\i View} source. This may not be to everybody's tastes. However, while this design limits the power of the template language, it does not limit the power or flexibility of the template system. This system supports arbitrarily complex text formatting.
Finally, {\i Mustache} is designed with an eye towards efficiency. Template instantiation is very quick, with an eye towards minimizing both memory use and memory fragmentation. As a result, it sounds like a perfect template system for our {\i mORMot} framework.
:   Mustache principles
There are two main parts to the {\i Mustache} template system:
- Templates (which are plain text files);
- Data dictionaries (aka {\i Context}).
For instance, given the following template:
$$<h1>{{header}}</h1>
$$
$${{#items}}
$$  {{#first}}
$$    <li><strong>{{name}}</strong></li>
$$  {{/first}}
$$  {{#link}}
$$    <li><a href="{{url}}">{{name}}</a></li>
$$  {{/link}}
$${{/items}}
$$
$${{#empty}}
$$  <p>The list is empty.</p>
$${{/empty}}
and the following data context:
#{
#  "header": "Colors",
#  "items": [
#      {"name": "red", "first": true, "url": "#Red"},
#      {"name": "green", "link": true, "url": "#Green"},
#      {"name": "blue", "link": true, "url": "#Blue"}
#  ],
#  "empty": true
#}
The {\i Mustache} engine will render this data as such:
$$<h1>Colors</h1>
$$<li><strong>red</strong></li>
$$<li><a href="#Green">green</a></li>
$$<li><a href="#Blue">blue</a></li>
$$<p>The list is empty.</p>
In fact, you did not see any "{\f1\fs20 if}" nor "{\i for}" loop in the template, but {\i Mustache} conventions make it easy to render the supplied data as the expected HTML output. It is up to the MVC {\i Controller} to render the data as expected by the template, e.g. for formatting dates or currency values.
:   Mustache templates
The {\i Mustache} template logic-less language has five types of tags:
- Variables;
- Sections;
- Inverted Sections;
- Comments;
- Partials.
All those tags will be identified with mustaches, i.e. {\f1\fs20 \{\{...\}\}}. Anything found in a template of this form is interpreted as a template marker. All other text is considered formatting text and is output verbatim at template expansion time.
|%20%80
|\b Marker|Description\b0
|{\f1\fs20 \{\{variable\}\}}|The {\f1\fs20 variable} name will be searched recursively within the current context (possibly with dotted names), and, if found, will be written as escaped HTML.\line If there is no such key, nothing will be rendered.
|{\f1\fs20 \{\{\{variable\}\}\}\line {\f1\fs20 \{\{& variable\}\}}}|The {\f1\fs20 variable} name will be searched recursively within the current context, and, if found, will be written directly, {\i without any HTML escape}.\line If there is no such key, nothing will be rendered.
|{\f1\fs20 \{\{#section\}\}}\line ...\line {\f1\fs20 \{\{/section\}\}}|Defines a block of text, aka {\i section}, which will be rendered depending of the {\f1\fs20 section} variable value, as searched in the current context:\line - If {\f1\fs20 section} equals {\f1\fs20 false} or is an {\i empty list} {\f1\fs20 []}, the whole block won't be rendered;\line - If {\f1\fs20 section} is non-{\f1\fs20 false} but not a list, it will be used as the context for a single rendering of the block;\line - If {\f1\fs20 section} is a non-empty list, the text in the block will be rendered once for each item in the list - the context of the block will be set to the current item for each iteration.
|{\f1\fs20 \{\{^section\}\}}\line ...\line {\f1\fs20 \{\{/section\}\}}|Defines a block of text, aka {\i inverted section}, which will be rendered depending of the {\f1\fs20 section} variable {\i inverted }value, as searched in the current context:\line - If {\f1\fs20 section} equals {\f1\fs20 false} or is an {\i empty list}, the whole block {\i will} be rendered;\line - If {\f1\fs20 section} is non-{\f1\fs20 false} or a non-empty list, it won't be rendered.
|{\f1\fs20 \{\{! comment\}\}}|The comment text will just be ignored.
|{\f1\fs20 \{\{>partial\}\}}|The {\f1\fs20 partial} name will be searched within the registered {\i partials list}, then will be executed at run-time (so recursive partials are possible), with the current execution context.
|{\f1\fs20 \{\{=...=\}\}}|The delimiters (i.e. by default {\f1\fs20 \{\{...\}\}}) will be replaced by the specified characters (may be convenient when double-braces may appear in the text).
|%
In addition to those standard markers, the {\i mORMot} implementation of {\i Mustache} features:
|%20%80
|\b Marker|Description\b0
|{\f1\fs20 \{\{helperName value\}\}}|{\i @*Expression Helper@}, able to change the value on the fly, before rendering. It could be used e.g. to display dates as text from {\f1\fs20 @*TDateTime@} or {\f1\fs20 @*TTimeLog@} values.
|{\f1\fs20 \{\{.\}\}}|This pseudo-variable refers to the context object itself instead of one of its members. This is particularly useful when iterating over lists.
|{\f1\fs20 \{\{-index\}\}}|This pseudo-variable returns the current item number when iterating over lists, starting counting at 1 ({\f1\fs20 \{\{-index0\}\}} will start counting at 0)
|{\f1\fs20 \{\{#-first\}\}}\line ...\line {\f1\fs20 \{\{/-first\}\}}|Defines a block of text (pseudo-section), which will be rendered - or {\i not} rendered for inverted {\f1\fs20 \{\{^-first\}\}} - for the {\i first} item when iterating over lists
|{\f1\fs20 \{\{#-last\}\}}\line ...\line {\f1\fs20 \{\{/-last\}\}}|Defines a block of text (pseudo-section), which will be rendered - or {\i not} rendered for inverted {\f1\fs20 \{\{^-last\}\}} - for the {\i last} item when iterating over lists
|{\f1\fs20 \{\{#-odd\}\}}\line ...\line {\f1\fs20 \{\{/-odd\}\}}|Defines a block of text (pseudo-section), which will be rendered - or {\i not} rendered for inverted {\f1\fs20 \{\{^-odd\}\}} - for the {\i odd} item number when iterating over lists: it can be very useful e.g. to display a list with alternating row colors
|{\f1\fs20 \{\{<partial\}\}}\line ...\line {\f1\fs20 \{\{/partial\}\}}|Defines an in-lined {\i partial} - to be called later via {\f1\fs20 \{\{>partial\}\}} - within the scope of the current template
|{\f1\fs20 \{\{"some text\}\}}|This pseudo-variable will supply the given text to a callback, which will for instance transform its content (e.g. translate it), before writing it to the output
|%
This set of markers will allow to easily write any kind of content, without any explicit logic nor nested code. As a major benefit, the template content could be edited and verified without the need of any {\i Mustache} compiler, since all those {\f1\fs20 \{\{...\}\}} markers will identify very clearly the resulting layout.
:    Variables
A typical Mustache template:
$Hello {{name}}
$You have just won {{value}} dollars!
$Well, {{taxed_value}} dollars, after taxes.
Given the following hash:
#{
#  "name": "Chris",
#  "value": 10000,
#  "taxed_value": 6000
#}
Will produce the following:
$Hello Chris
$You have just won 10000 dollars!
$Well, 6000 dollars, after taxes.
You can note that {\f1\fs20 \{\{variable\}\}} tags are escaped for HTML by default. This is a mandatory security feature. In fact, all @*web application@s which create HTML documents can be vulnerable to Cross-Site-Scripting (XSS) attacks unless data inserted into a template is appropriately sanitized and/or escaped. With Mustache, this is done by default. Of course, you can override it and force to {\i not-escape} the value, using {\f1\fs20 \{\{\{variable\}\}\} or {\f1\fs20 \{\{& variable\}\}}}.
For instance:
|%24%38%38
|\b Template|Context|Output\b0
|{\f1\fs20 * \{\{name\}\}\line * \{\{age\}\}\line * \{\{company\}\}\line * \{\{\{company\}\}\}|\{\line   "name": "Chris",\line   "company": "<b>GitHub</b>"\line \}|* Chris\line *\line * &lt;b&gt;GitHub&lt;/b&gt;\line * <b>GitHub</b>}
|%
Variables resolve names within the current context with an optional dotted syntax, for instance:
|%29%37%36
|\b Template|Context|Output\b0
|{\f1\fs20 * \{\{people.name\}\}\line * \{\{people.age\}\}\line * \{\{people.company\}\}\line * \{\{\{people.company\}\}\}|\{\line  "people": \{\line   "name":"Chris",\line   "company":"<b>GitHub</b>"\line  \}\line \}|* Chris\line *\line * &lt;b&gt;GitHub&lt;/b&gt;\line * <b>GitHub</b>}
|%
:    Sections
{\i Sections} render blocks of text one or more times, depending on the value of the key in the current context.
In our "wining template" above, what happen if we do want to hide the tax details?\line In most script languages, we may write an {\f1\fs20 if ... } block within the template. This is what {\i Mustache} avoids. So we define a section, which will be rendered on need.
The template becomes:
$Hello {{name}}
$You have just won {{value}} dollars!
${{#in_ca}}
$Well, {{taxed_value}} dollars, after taxes.
${{/in_ca}}
Here, we created a new section, named {\f1\fs20 in_ca}.
Given the hash value of {\f1\fs20 in_ca} (and its presence), the section will be rendered, or not:
|%40%55
|\b Context|Output\b0
|{\f1\fs20 \{\line   "name": "Chris",\line   "value": 10000,\line   "taxed_value": 6000,\line   "in_ca": true\line \}|Hello Chris\line You have just won 10000 dollars!\line Well, 6000 dollars, after taxes.}
|{\f1\fs20 \{\line   "name": "Chris",\line   "value": 10000,\line   "taxed_value": 6000,\line   "in_ca": false\line \}|Hello Chris\line You have just won 10000 dollars!}
|{\f1\fs20 \{\line   "name": "Chris",\line   "value": 10000,\line   "taxed_value": 6000\line \}|Hello Chris\line You have just won 10000 dollars!}
|%
Sections also change the context of its inner block. It means that the section variable content becomes the top-most context which will be used to identify any supplied variable key.
Therefore, the following context will be perfectly valid: we can define {\f1\fs20 taxed_value} as a member of {\f1\fs20 in_ca}, and it will be rendered directly, since it is part of the new context.
|%40%55
|\b Context|Output\b0
|{\f1\fs20 \{\line   "name": "Chris",\line   "value": 10000,\line   "in_ca": \{\line      "taxed_value": 6000\line   \}\line \}|Hello Chris\line You have just won 10000 dollars!\line Well, 6000 dollars, after taxes.}
|{\f1\fs20 \{\line   "name": "Chris",\line   "value": 10000,\line   "taxed_value": 6000\line \}|Hello Chris\line You have just won 10000 dollars!}
|{\f1\fs20 \{\line   "name": "Chris",\line   "value": 10000,\line   "taxed_value": 3000,\line   "in_ca": \{\line      "taxed_value": 6000\line   \}\line \}|Hello Chris\line You have just won 10000 dollars!\line Well, 6000 dollars, after taxes.}
|%
In the latest context above, there are two {\f1\fs20 taxed_value} variables. The engine will use the one defined by the context in the {\f1\fs20 in_ca} section, i.e. {\f1\fs20 in_ca.taxed_value}; the one defined at the root context level (which equals 3000) is just ignored.
If the variable pointed by the section name is a list, the text in the block will be rendered once for each item in the list. The context of the block will be set to the current item for each iteration.\line In this way we can loop over collections. {\i Mustache} allows any depth of nested loops (e.g. any level of master/details information).
|%24%38%38
|\b Template|Context|Output\b0
|{\f1\fs20 \{\{#repo\}\}\line  <b>\{\{name\}\}</b>\line \{\{/repo\}\}|\{\line   "repo": [\line     { "name": "resque" },\line     { "name": "hub" },\line     { "name": "rip" }\line   ]\line \}|<b>resque</b>\line <b>hub</b>\line <b>rip</b>}
|{\f1\fs20 \{\{#repo\}\}\line  <b>\{\{.\}\}</b>\line \{\{/repo\}\}|\{\line   "repo":\line      ["resque", "hub", "rip"]\line \}|<b>resque</b>\line <b>hub</b>\line <b>rip</b>}
|%
The latest template makes use of the {\f1\fs20 \{\{.\}\}} pseudo-variable, which allows to render the current item of the list.
:    Inverted Sections
An inverted section begins with a caret ({\f1\fs20 ^}) and ends as a standard (non-inverted) section. They may render text once, based on the {\i inverse} value of the key. That is, the text block will be rendered if the key doesn't exist, is false, or is an empty list.
Inverted sections are usually defined after a standard section, to render some message in case no information will be written in the non-inverted section:
|%30%30%30
|\b Template|Context|Output\b0
|{\f1\fs20 \{\{#repo\}\}\line  <b>\{\{.\}\}</b>\line \{\{/repo\}\}\line \{\{^repo\}\}\line No repos :(\line \{\{/repo\}\}|\{\line   "repo":\line      []\line \}|No repos :(}
|%
:    Partials
Partials are some kind of external sub-templates which can be included within a main template, for instance to follow the same rendering at several places. Just like functions in code, they do ease template maintainability and spare development time.
Partials are rendered at runtime (as opposed to compile time), so recursive partials are possible. Just avoid infinite loops. They also inherit the calling context, so can easily be re-used within a list section, or together with plain variables.
In practice, partials shall be supplied together with the data context - they could be seen as "template context".
For example, this "main" template uses a {\f1\fs20 \{\{> user\}\}} partial:
$$<h2>Names</h2>
$${{#names}}
$$  {{> user}}
$${{/names}}
With the following template registered as "user":
$$<strong>{{name}}</strong>
Can be thought of as a single, expanded template:
$$<h2>Names</h2>
$${{#names}}
$$  <strong>{{name}}</strong>
$${{/names}}
In {\i mORMot}'s implementations, you can also create some {\i internal} partials, defined as {\f1\fs20 \{\{<partial\}\} ... \{\{/partial\}\}} pseudo-sections. It may decrease the need of maintaining multiple template files, and refine the rendering layout.
For instance, the previous template may be defined at once:
$$<h2>Names</h2>
$${{#names}}
$$  {{>user}}
$${{/names}}
$$
$${{<user}}
$$<strong>{{name}}</strong>
$${{/user}}
The same file will define both the partial and the main template. Note that we defined the internal partial after the main template, but we may have defined it anywhere in the main template logic: internal partials definitions are ignored when rendering the main template, just like comments.
:   SynMustache unit
Part of our {\i mORMot} framework, we implemented an optimized {\i Mustache} template engine in the {\f1\fs20 SynMustache} unit:
- It is the first {\i Delphi} implementation of {\i Mustache};
- It has a separate parser and renderer (so you can compile your templates ahead of time);
- The parser features a shared cache of compiled templates;
- It passes all official {\i Mustache} specification tests, as defined at @http://github.com/mustache/spec - including all weird whitespace process;
- External partials can be supplied as {\f1\fs20 TSynMustachePartials} dictionaries;
- {\f1\fs20 \{\{.\}\}}, {\f1\fs20 \{\{-index\}\}} and {\f1\fs20 \{\{"some text\}\}} pseudo-variables were added to the standard {\i Mustache} syntax;
- {\f1\fs20 \{\{#-first\}\}}, {\f1\fs20 \{\{#-last\}\}} and {\f1\fs20 \{\{#-odd\}\}} pseudo-sections were added to the standard {\i Mustache} syntax;
- Internal partials can be defined via {\f1\fs20 \{\{<partial\}\}} - also a nice addition to the standard {\i Mustache} syntax;
- It allows the data context to be supplied as JSON or our @80@;
- Almost no memory allocation is performed during the rendering;
- It is natively @*UTF-8@, from the ground up, with optimized conversion of any string data;
- Performance has been tuned and grounded in {\f1\fs20 SynCommons.pas}'s optimized code;
- Each parsed template is thread-safe and re-entrant;
- It follows the {\i Open/Closed principle} - see @47@ - so that any aspect of the process can be customized and extended (e.g. for any kind of data context);
- It is perfectly integrated with the other bricks of our {\i mORMot} framework, ready to implement dynamic web sites with true @10@ design, and full separation of concerns in the views written in {\i Mustache}, the controllers being e.g. interface-based services - see @63@, and the models being our @13@ classes;
- API is flexible and easy to use.
:    Variables
Now, let's see some code.\line First, we define our needed variables:
!var mustache: TSynMustache;
!    doc: variant;
In order to parse a template, you just need to call:
!  mustache := TSynMustache.Parse(
!    'Hello {{name}}'#13#10'You have just won {{value}} dollars!');
It will return a compiled instance of the template.\line The {\f1\fs20 Parse()} class method will use the shared cache, so you won't need to release the {\f1\fs20 mustache} instance once you are done with it: no need to write a {\f1\fs20 try ... finally mustache.Free; end} block.
You can use a {\f1\fs20 TDocVariant} to supply the context data (with late-binding):
!  TDocVariant.New(doc);
!  doc.name := 'Chris';
!  doc.value := 10000;
As an alternative, you may have defined the context data as such:
!  doc := _ObjFast(['name','Chris','value',1000]);
Now you can render the template with this context:
!  html := mustache.Render(doc);
!  // now html='Hello Chris'#13#10'You have just won 10000 dollars!'
If you want to supply the context data as JSON, then render it, you may write:
!  mustache := TSynMustache.Parse(
!    'Hello {{value.name}}'#13#10'You have just won {{value.value}} dollars!');
!  html := mustache.RenderJSON('{value:{name:"Chris",value:10000}}');
!  // now html='Hello Chris'#13#10'You have just won 10000 dollars!'
Note that here, the JSON is supplied with {\i MongoDB}-like @*extended syntax@ (i.e. field names are unquoted), and that {\f1\fs20 TSynMustache} is able to identify a dotted-named variable within the execution context.
As an alternative, you could use the following syntax to create the data context as JSON, with a set of parameters, therefore easier to work with in real code storing data in variables (for instance, any {\f1\fs20 string} variable is quoted as expected by JSON, and converted into @*UTF-8@):
!  mustache := TSynMustache.Parse(
!    'Hello {{name}}'#13#10'You have just won {{value}} dollars!');
!  html := mustache.RenderJSON('{name:?,value:?}',[],['Chris',10000]);
!  html='Hello Chris'#13#10'You have just won 10000 dollars!'
You can find in the {\f1\fs20 mORMot.pas} unit the {\f1\fs20 ObjectToJSON()} function which is able to transform any {\f1\fs20 TPersistent} instance into valid JSON content, ready to be supplied to a {\f1\fs20 TSynMustache} compiled instance.\line If the object's published properties have some getter functions, they will be called on the fly to process the data (e.g. returning 'FirstName Name' as FullName by concatenating both sub-fields).
:    Sections
Sections are handled as expected:
!  mustache := TSynMustache.Parse('Shown.{{#person}}As {{name}}!{{/person}}end{{name}}');
!  html := mustache.RenderJSON('{person:{age:?,name:?}}',[10,'toto']);
!  // now html='Shown.As toto!end'
Note that the sections change the data context, so that within the {\f1\fs20 #person} section, you can directly access to the data context {\f1\fs20 person} member, i.e. writing directly {\f1\fs20 \{\{name\}\}}
It supports also inverted sections:
!  mustache := TSynMustache.Parse('Shown.{{^person}}Never shown!{{/person}}end');
!  html := mustache.RenderJSON('{person:true}');
!  // now html='Shown.end'
To render a list of items, you can write for instance (using the {\f1\fs20 \{\{.\}\}} pseudo-variable):
!  mustache := TSynMustache.Parse('{{#things}}{{.}}{{/things}}');
!  html := mustache.RenderJSON('{things:["one", "two", "three"]}');
!  // now html='onetwothree'
The {\f1\fs20 \{\{-index\}\}} pseudo-variable allows to numerate the list items, when rendering:
!  mustache := TSynMustache.Parse(
!    'My favorite things:'#$A'{{#things}}{{-index}}. {{.}}'#$A'{{/things}}');
!  html := mustache.RenderJSON('{things:["Peanut butter", "Pen spinning", "Handstands"]}');
!  // now html='My favorite things:'#$A'1. Peanut butter'#$A'2. Pen spinning'#$A+
!  //          '3. Handstands'#$A,'-index pseudo variable'
:    Partials
External partials (i.e. standard {\i Mustache} partials) can be defined using {\f1\fs20 TSynMustachePartials}. You can define and maintain a list of {\f1\fs20 TSynMustachePartials} instances, or you can use a one-time partial, for a given rendering process, as such:
!  mustache := TSynMustache.Parse('{{>partial}}'#$A'3');
!  html := mustache.RenderJSON('{}',TSynMustachePartials.CreateOwned(['partial','1'#$A'2']));
!  // now html='1'#$A'23','external partials'
Here {\f1\fs20 TSynMustachePartials.CreateOwned()} expects the partials to be supplied as name/value pairs.
Internal partials (one of the {\f1\fs20 SynMustache} extensions), can be defined directly in the main template:
!  mustache := TSynMustache.Parse('{{<partial}}1'#$A'2{{name}}{{/partial}}{{>partial}}4');
!  html := mustache.RenderJSON('{name:3}');
!  // now html='1'#$A'234','internal partials'
:    Expression Helpers
{\i @**Expression Helper@s} are an extension to the standard {\i Mustache} definition. They allow to define your own set of functions which will be called during the rendering, to transform one value from the context into a value to be rendered.
{\f1\fs20 TSynMustache.HelpersGetStandardList} will return a list of standard static helpers, able to convert {\f1\fs20 @*TDateTime@} or {\f1\fs20 @*TTimeLog@} values into text, or convert any value into its @*JSON@ representation. The current list of registered helpers are {\f1\fs20 DateTimeToText}, {\f1\fs20 DateToText}, {\f1\fs20 DateFmt}, {\f1\fs20 TimeLogToText}, {\f1\fs20 BlobToBase64}, {\f1\fs20 JSONQuote}, {\f1\fs20 JSONQuoteURI}, {\f1\fs20 ToJSON}, {\f1\fs20 EnumTrim}, {\f1\fs20 EnumTrimRight}, {\f1\fs20 PowerOfTwo} , {\f1\fs20 Equals}, {\f1\fs20 If}, {\f1\fs20 MarkdownToHtml}, {\f1\fs20 SimpleToHtml} and {\f1\fs20 WikiToHtml}. For instance, {\f1\fs20 \{\{TimeLogToText CreatedAt\}\}} will convert a {\f1\fs20 TCreateTime} field value into ready-to-be-displayed text.
The mustache tag syntax is  {\f1\fs20 \{\{helpername value\}\}}. The supplied {\f1\fs20 value} parameter may be a variable name in the current context, or could be a constant number ({\f1\fs20 \{\{helpername 123\}\}}), a constant JSON string ({\f1\fs20 \{\{helpername "constant text"\}\}}), a JSON array ({\f1\fs20 \{\{helpername [1,2,3]\}\}}) or a JSON object ({\f1\fs20 \{\{helpername \{name:"john",age:24\}\}\}}). The value could be also a comma-seperated set of values, which will be translated into a corresponding JSON array, the values being extracted from the current context, as with {\f1\fs20 \{\{DateFmt DateValue,"dd/mm/yyy"\}\}}.
You could call recursively the helpers, just like you nest functions: {\f1\fs20 \{\{helper1 helper2 value\}\}} will call {\f1\fs20 helper2} with the supplied {\f1\fs20 value}, which result will be passed as value to {\f1\fs20 helper1}.
But you can create your own list of registered {\i Expression Helpers}, even including some business logic, to compute any data during rendering, via {\f1\fs20 TSynMustache.HelperAdd} methods.
The helper should be implemented with such a method:
!class procedure TSynMustache.JSONQuote(const Value: variant; out result: variant);
!var json: RawUTF8;
!begin
!  QuotedStrJSON(VariantToUTF8(Value),json);
!  RawUTF8ToVariant(json,result);
!end;
Here, the supplied {\f1\fs20 Value} parameter will be either from a variable of the context, or a constant, from JSON number, string, array or object - encoded as a @80@.
If the parameters were supplied as a comma-separated list, you may write multi-parameter functions as such:
!class procedure TSynMustache.DateFmt(const Value: variant; out result: variant);
!begin
!  with _Safe(Value)^ do
!    if (Kind=dvArray) and (Count=2) and (TVarData(Values[0]).VType=varDate) then
!      result := FormatDateTime(Values[1],TVarData(Values[0]).VDate) else
!      SetVariantNull(result);
!end;
So you could use such expression helper this way:
$ La date courante en France est : {{DateFmt DateValue,"dd/mm/yyyy"}}
The {\f1\fs20 Equals} helper is defined as such:
!class procedure TSynMustache.Equals_(const Value: variant; out result: variant);
!begin // {{#Equals .,12}}
!  with _Safe(Value)^ do
!    if (Kind=dvArray) and (Count=2) and
!       (SortDynArrayVariant(Values[0],Values[1])=0) then
!      result := true else
!      SetVariantNull(result);
!end;
You may use it in your template to provide additional view logic:
${{#Equals Category,"Admin"}}
$Welcome, Mister Administrator!
${{/Equals Category,"Admin"}}
The section ending may optionally only contain the helper name, so the following syntax is also correct, and perhaps less error-prone:
${{#Equals Category,"Admin"}}
$Welcome, Mister Administrator!
${{/Equals}}
The {\f1\fs20 #If} helper is even more powerful, since it allows to define some view logic, via {\f1\fs20 = < > <= >= <>} operators set between two values:
${{#if .,"=",6}} Welcome, number six! {{/if}}
${{#if Total,">",1000}} Thanks for your income: your loyalty will be rewarded. {{/if}}
${{#if info,"<>",""}} Warning: {{info}} {{/if}}
As an alternative, you could just put the operator without a string parameter:
${{#if .=6}} Welcome, number six! {{/if}}
${{#if Total>1000}} Thanks for your income: your loyalty will be rewarded. {{/if}}
${{#if info<>""}} Warning: {{info}} {{/if}}
This latest syntax may it pretty convenient to work with. Of course, since {\i Mustache} is expected to be a {\i logic-less} templating engine, you should better not use the {\f1\fs20 #if} helper in most cases, but rather add some dedicated flags in the supplied data context:
${{#isNumber6}} Welcome, number six! {{/isNumber6}}
${{#showLoyaltyMessage}} Thanks for your income: your loyalty will be rewarded. {{/#showLoyaltyMessage}}
${{#showWarning}} Warning: {{info}} {{/#showWarning}}
Helpers can be used to convert some {\i @*wiki@} or {\i @*markdown@} content into plain HTML, for instance, in the MVC blog sample, a {\f1\fs20 ContentHtml} boolean flag defines if a content (here the {\f1\fs20 abstract} text field) is already HTML-encoded, or if it needs to be converted via the {\f1\fs20 WikiToHtml} helper:
${{#ContentHtml}}{{{abstract}}}{{/ContentHtml}}{{^ContentHtml}}{{{WikiToHtml abstract}}}{{/ContentHtml}}
The framework also offers some built-in optional {\i Helpers} tied to its @*ORM@, if you create a MVC @*web application@ using {\f1\fs20 mORMotMVC.pas} - see @108@ - you can register a set of {\i Expression Helpers} to let your {\i Mustache} view retrieve a given {\f1\fs20 TSQLRecord}, from its ID, or display a given instance fields in an auto-generated table.
For instance, you may write:
! aMVCMustacheView.RegisterExpressionHelpersForTables(aRestServer,[TSQLMyRecord]);
This will define two {\i Expression Helpers} for the specified table:
- Any {\f1\fs20 \{\{#TSQLMyRecord MyRecordID\}\}} ... {\f1\fs20 \{\{/TSQLMyRecord MyRecordID\}\}} {\i Mustache} tag will read a {\f1\fs20 TSQLMyRecord} from the supplied {\f1\fs20 ID} value and put its fields in the current rendering data context, ready to be displayed in the view.
- Any {\f1\fs20 \{\{TSQLMyRecord.HtmlTable MyRecord\}\}} {\i Mustache} tag which will create a HTML table containing all information about the supplied {\f1\fs20 MyRecord} fields (from the current data context), with complex field handling (like {\f1\fs20 @*TDateTime@}, {\f1\fs20 @*TTimeLog@}, sets or enumerations), and proper display of the field names (and {\i @*i18n@}).
:    Internationalization
You can define {\f1\fs20 \{\{"some text\}\}} pseudo-variables in your templates, which text will be supplied to a callback, ready to be transformed on the fly: it may be convenient for @*i18n@ of web applications.
By default, the text will be written directly to the output buffer, but you can define a callback which may be used e.g. for text translation:
!procedure TTestLowLevelTypes.MustacheTranslate(var English: string);
!begin
!  if English='Hello' then
!    English := 'Bonjour' else
!  if English='You have just won' then
!    English := 'Vous venez de gagner';
!end;
Of course, in a real application, you may assign one {\f1\fs20 TLanguageFile.Translate(var English: string)} method, as defined in the {\f1\fs20 mORMoti18n.pas} unit.
Then, you will be able to define your template as such:
!  mustache := TSynMustache.Parse(
!    '{{"Hello}} {{name}}'#13#10'{{"You have just won}} {{value}} {{"dollars}}!');
!  html := mustache.RenderJSON('{name:?,value:?}',[],['Chris',10000],nil,MustacheTranslate);
!  // now html='Bonjour Chris'#$D#$A'Vous venez de gagner 10000 dollars!'
All text has indeed been translated as expected.
:   Low-level integration with method-based services
You can easily integrate the {\i @*Mustache@} template engine with the framework's @*ORM@. To avoid any unneeded temporary conversion, you can use the {\f1\fs20 TSQLRest.RetrieveDocVariantArray()} method, and provide its {\f1\fs20 TDocVariant} result as the data context of {\f1\fs20 TSynMustache.Render()}.
For instance, you may write, in any method-based service - see @49@:
!var template: TSynMustache;
!    html: RawUTF8;
! ...
!  template := TSynMustache.Parse(
!    '<ul>{{#items}}<li>{{Name}} was born on {{BirthDate}}</li>{{/items}}</ul>');
!  html := template.Render(
!    aClient.RetrieveDocVariantArray(TSQLBaby,'items','Name,BirthDate'));
!  // now html will contain a ready-to-be-displayed unordered list
Of course, this {\f1\fs20 TSQLRest.RetrieveDocVariantArray()} method accepts an optional WHERE clause, to be used according to your needs. You may even use paging, to split the list in smaller pieces.
Following this low-level method-based services process, you can easily create a high performance web server using {\i mORMot}, following the @*MVC@ pattern as such:
|%20%80
|\b MVC|mORMot\b0
|{\i Model}|@13@ and its {\f1\fs20 TSQLModel} / {\f1\fs20 TSQLRecord} definitions
|{\i View}|@81@\line (may be stored as separated files or within the database)
|{\i Controller}|Method-based services - see @49@
|%
But still, a lot of code is needed to glue the MVC parts.
:   MVC/MVVM Design
In practice, the method-based services @*MVC@ pattern is difficult to work with. You have a lot of plumbing to code by yourself, e.g. parameter marshalling, rendering or routing.
The {\f1\fs20 mORMotMVC.pas} unit offers a true @**MVVM@ ({\i Model View ViewModel})design, much more advanced, which relies on {\f1\fs20 interface} definitions to build the application - see @46@:
|%20%80
|\b MVVM|mORMot\b0
|{\i Model}|@13@ and its {\f1\fs20 TSQLModel} / {\f1\fs20 TSQLRecord} definitions
|{\i View}|@81@\line (may be stored as separated files or within the database)
|{\i ViewModel}|{\f1\fs20 Interface} services - see also @63@
|%
In the MVVM pattern, both {\i Model} and {\i View} components do match the classic @10@ layout. But the {\i ViewModel} will define some kind of "model for the view", i.e. the data context to be sent and retrieved from the view.
In the {\i mORMot} implementation, {\f1\fs20 interface} methods are used to define the execution context of any request, following the {\i @*convention over configuration@} pattern of our framework.\line In fact, the following conventions are used to define the {\i ViewModel}:
|%23%77
|\b ViewModel|mORMot\b0
|{\i Route}|From the {\f1\fs20 interface} name and its method name
|{\i Command}|Defined by the method name
|{\i Controller}|Defined by the method implementation
|{\i ViewModel Context}|Transmitted {\i by representation}, as @*JSON@, including complex values like {\f1\fs20 @*TSQLRecord@}, records, @*dynamic array@s or @*variant@s (including {\f1\fs20 @*TDocVariant@})
|{\i Input Context}|Transmitted as method input parameters ({\f1\fs20 const}/{\f1\fs20 var}) from the {\i View}
|{\i Output Context}|Method output parameters ({\f1\fs20 var}/{\f1\fs20 out}) are sent to the {\i View}
|{\i Actions}|A method will render the associated view with the output parameters, or go to another command (optionally via {\f1\fs20 EMVCApplication})
|%
This may sound pretty unusual (if you are coming from a {\i RubyOnRails}, {\i AngularJS}, {\i Meteor} or {\i .Net} implementations), but it has been identified to be pretty convenient to use. Main benefit is that you do not need to define explicit data structures for the {\i ViewModel} layer. The method parameters will declare the execution context for you at {\f1\fs20 interface} level, ready to be implemented in a {\f1\fs20 TMVCApplication} class. In practice, this implementation uses the {\f1\fs20 interface} input and output parameters are an alternate way to define the {\f1\fs20 $scope} content of an {\i AngularJS} application.
The fact that the {\i ViewModel} data context is transmitted as JSON content - {\i by representation} just like @*REST@ - see @9@ - allows nice side effects:
- {\i Views} do not know anything about the execution context, so are very likely to be uncoupled from any business logic - this will enhance security and maintainability of your applications;
- You can optionally see in real time the JSON data context (by using a fake {\f1\fs20 root/methodname{\f1\fs20 /json}} URI) of a running application, for easier debugging of the {\i Controller} or the {\i Views};
- You can test any {\i View} by using fake static JSON content, without the need of a real server;
- In fact, {\i Views} could be even not tied to the web model, but run in a classic rich application, with VCL/FMX User Interface (we still need to automate the binding process to UI components, but this is technically feasible, whereas almost no other MVC web framework do support this);
- Since {\f1\fs20 interface} are used to define the {\i Controller}, you could @*mock@ and @*stub@ them - see @62@ - for proper unit testing;
- In the {\i Controller} code, you have access to the {\i mORMot} ORM methods and services to write the various commands, making it pretty easy to implement a web front-end to any @*SOA@ project (also sharing a lot of high-level {\i Domain} types);
- The associated data {\i Model} is {\i mORMot}'s ORM, which is also optimized for JSON processing, so most of memory fragmentation is reduced to the minimum during the rendering (see e.g. the use of {\f1\fs20 RawJSON} below);
- The {\i Controller} will be most of the time hosted within the web server application, but {\i may} be physically hosted in another remote process - this remote {\i Controller} service may even be shared between web and VCL/FMX clients;
- Several levels of @*cache@ could be implemented, based on the JSON content, to leverage the server resources and scale over a huge number of clients.
The next chapter will uncover how to build such solid MVC / MVVM @*Web Application@s using {\i mORMot}.
:108MVC/MVVM Web applications
%cartoon01.png
We will now explain how to build a @*MVC@/@*MVVM@ @**web application@ using {\i mORMot}, starting from the "{\i 30 - MVC Server}" sample. Following explanations may be a bit unsynchronized from the current state of the sample source code in the "unstable" branch of the framework repository, but you will get here below the main intangible points.
This little web application publishes a simple BLOG, not fully finished yet (this is a {\i Sample}, remember!). But you can still execute it in your desktop browser, or any mobile device (thanks to a simple {\i @*Bootstrap@}-based @*responsive design@), and see the articles list, view one article and its comments, view the author information, log in and out.
This sample is implemented as such:
|%15%22%63
|\b MVVM|Source|mORMot\b0
|{\i Model}|{\f1\fs20 MVCModel.pas}|{\f1\fs20 TSQLRestServerDB} ORM over a {\i SQlite3} database
|{\i View}|{\f1\fs20 *.html}|@81@ in the {\i Views} sub-folder
|{\i ViewModel}|{\f1\fs20 MVCViewModel.pas}|Defined as one {\f1\fs20 IBlogApplication} interface
|%
For the sake of the simplicity, the sample will create some fake data in its own local {\i SQlite3} database, the first time it is executed.
: MVCModel
:  Data Model
The {\f1\fs20 MVCModel.pas} unit defines the database {\i Model}, as regular {\f1\fs20 TSQLRecord} classes.\line For instance, you will find the following type definitions:
!  TSQLContent = class(TSQLRecordTimestamped)
!  private ...
!  published
!    property Title: RawUTF8 index 80 read fTitle write fTitle;
!    property Content: RawUTF8 read fContent write fContent;
!    property Author: TSQLAuthor read fAuthor write fAuthor;
!    property AuthorName: RawUTF8 index 50 read fAuthorName write fAuthorName;
!  end;
!
!  TSQLArticle = class(TSQLContent)
!  private ...
!  public
!    class function CurrentPublishedMonth: Integer;
!    class procedure InitializeTable(Server: TSQLRestServer; const FieldName: RawUTF8;
!      Options: TSQLInitializeTableOptions); override;
!    procedure TagsAddOrdered(aTagID: integer; var aTags: TSQLTags);
!  published
!    property PublishedMonth: Integer read fPublishedMonth write fPublishedMonth;
!    property Abstract: RawUTF8 index 1024 read fAbstract write fAbstract;
!    property Tags: TIntegerDynArray index 1 read fTags write fTags;
!  end;
!
!  TSQLComment = class(TSQLContent)
!  private ...
!  published
!    property Article: TSQLArticle read fArticle write fArticle;
!  end;
Then the whole database model will be created in this function:
!function CreateModel: TSQLModel;
!begin
!  result := TSQLModel.Create([TSQLBlogInfo,TSQLAuthor,
!    TSQLTag,TSQLArticle,TSQLComment,TSQLArticleSearch],'blog');
!  TSQLArticle.AddFilterNotVoidText(['Title','Content']);
!  TSQLComment.AddFilterNotVoidText(['Title','Content']);
!  TSQLTag.AddFilterNotVoidText(['Ident']);
!  result.Props[TSQLArticleSearch].FTS4WithoutContent(TSQLArticle);
!end;
As you can discover:
- We used {\f1\fs20 class} inheritance to gather properties for similar tables;
- Some classes are {\i not} part of the model, since they are just {\f1\fs20 abstract} parents, e.g. {\f1\fs20 TSQLContent} is not part of the model, but {\f1\fs20 TSQLArticle} and {\f1\fs20 TSQLComment} are;
- We defined some regular {\i one-to-one} relationships, e.g. every {\f1\fs20 Content} (which may be either an {\f1\fs20 Article} or a {\f1\fs20 Comment}) will be tied to one {\f1\fs20 Author} - see @70@;
- We defined some regular {\i one-to-many} relationships, e.g. every {\f1\fs20 Comment} will be tied to one {\f1\fs20 Article};
- {\f1\fs20 Article} tags are stored as a dynamic array of integer within the record, and not in a separated pivot table: it will make the database smaller, and queries faster (since we avoid a JOIN);
- Some properties are defined (and stored) twice, e.g. {\f1\fs20 TSQLContent} defines one {\f1\fs20 AuthorName} field in addition to the {\f1\fs20 Author} ID field, as a convenient direct access to the author name, therefore avoiding a JOINed query at each {\f1\fs20 Article} or a {\f1\fs20 Comment} display - see @29@;
- We defined the maximum expected width for text fields (e.g. via {\f1\fs20 Title: RawUTF8 {\b @*index@ 80}}), even if it won't be used by {\i SQLite3} - it will ease any eventual migration to an external database, in the future - see @145@;
- Some validation rules are set using {\f1\fs20 TSQLArticle.AddFilterNotVoidText()} method, which will be applied before an article is stored in the controller's code (in {\f1\fs20 TBlogApplication. ArticleCommit});
- The whole application will run without writing any SQL, but just high-level ORM methods;
- Even if we want to avoid writing SQL, we tried to modelize the data to fit regular RDBMS expectations, e.g. for most used queries (like the one run from the main page of the BLOG);
- @*Full Text@ indexation data, implemented as @8@ in the {\i SQLite3} engine, is stored in a dedicated {\f1\fs20 TSQLArticleSearch} table - see @144@ for details about this powerful feature.
Foreign keys and indexes are managed as such:
- The {\f1\fs20 TSQLRecord.ID} @*primary key@ of any ORM class will be indexed;
- For both {\i one-to-one} and {\i one-to-many} relationships, indexes are created by the ORM: for instance, {\f1\fs20 TSQLArticle.Author} and {\f1\fs20 TSQLComment.Author} will be indexed, just as {\f1\fs20 TSQLComment.Article};
- A SQL index will be needed for {\f1\fs20 TSQLArticle.PublishedMonth} field, which is used to display a list of publication months in the main BLOG page, and link to the corresponding articles.\line The following code will take care of it:
!class procedure TSQLArticle.InitializeTable(Server: TSQLRestServer;
!  const FieldName: RawUTF8; Options: TSQLInitializeTableOptions);
!begin
!  inherited;
!  if (FieldName='') or (FieldName='PublishedMonth') then
!    Server.CreateSQLIndex(TSQLArticle,'PublishedMonth',false);
!end;
:  Hosted in a REST server over HTTP
The ORM is defined to run over a {\i SQLite3} database in the main {\f1\fs20 MVCServer.dpr} program, then served via a HTTP server as defined in {\f1\fs20 MVCServer.dpr}:
!  aModel := CreateModel;
!  try
!    aServer := TSQLRestServerDB.Create(aModel,ChangeFileExt(paramstr(0),'.db'));
!    try
!      aServer.DB.Synchronous := smNormal;
!      aServer.DB.LockingMode := lmExclusive;
!      aServer.CreateMissingTables;
!!      aApplication := TBlogApplication.Create;
!      try
!!        aApplication.Start(aServer);
!        aHTTPServer := TSQLHttpServer.Create('8092',aServer,'+',useHttpApiRegisteringURI);
!        try
!!          aHTTPServer.RootRedirectToURI('blog/default'); // redirect server:8092
!          writeln('"MVC Blog Server" launched on port 8092');
!  ....
In comparison to a regular @35@, we instantiated a {\f1\fs20 TBlogApplication}, which will {\i inject} the MVC behavior to {\f1\fs20 aServer} and {\f1\fs20 aHTTPServer}. The same {\i mORMot} program could be used as a @*REST@ful server for remote @13@ and @17@, and also for publishing a web application, sharing the same data and business logic, over a single HTTP URI and port.\line A call to {\f1\fs20 RootRedirectToURI()} will let any @http://server:8092 HTTP request be redirected to @http://server:8092/blog/default which is our BLOG application main page. The other URIs could be used as usual, as any {\i mORMot}'s @6@.
You could also use sub-domain hosting, as defined for @140@, to make a difference between the REST methods and the MVC web site. For instance, you may define some per domain / per sub-domain hosting redirection:
! aHttpServer.DomainHostRedirect('rest.project.com','root');      // 'root' is current Model.Root
! aHttpServer.DomainHostRedirect('project.com','root/html');      // call the Html() method
! aHttpServer.DomainHostRedirect('www.project.com','root/html');  // call the Html() method
! aHttpServer.DomainHostRedirect('blog.project.com','root/blog'); // MVC application
All ORM/SOA activity should be accessed remotely via {\f1\fs20 rest.project.com}, then will be handled as expected by the ORM/SOA methods of the {\f1\fs20 TSQLRestServer} instance.\line For proper AJAX / JavaScript process, you may have to write:
! aHttpServer.AccessControlAllowOrigin := '*'; // allow cross-site AJAX queries
Any attempt to access to the {\f1\fs20 project.com} or {\f1\fs20 www.project.com} URI will be redirected to the following method-based service:
!procedure TMyServer.Html(Ctxt: TSQLRestServerURIContext);
!begin
!  if fMyFileCache='' then
!    fMyFileCache := StringFromFile(ChangeFileExt(paramstr(0),'.html'));
!  Ctxt.Returns(fMyFileCache,HTTP_SUCCESS,HTML_CONTENT_TYPE_HEADER,true);
!end;
This method will serve some static HTML content as the main front end page of this server connected to the Internet. For best performance, this UTF-8 content is cached in memory, and the HTTP 304 command will be handled, if the browser supports it. Of course, your application may return some more complex content, even serving a set of files hosted in a local folder, e.g. by calling {\f1\fs20 Ctxt.ReturnFile()} or {\f1\fs20 Ctxt.ReturnFileFromFolder()} methods in this {\f1\fs20 Html()} service:
!procedure TMyServer.Html(Ctxt: TSQLRestServerURIContext);
!begin
!  Ctxt.ReturnFileFromFolder('c:\www');
!end;
This single method will search for any matching file in the local {\f1\fs20 c:\\www} folder and its sub-directories, returning the default {\f1\fs20 index.html} content if no file is specified at URI level. See the optional parameters to the {\f1\fs20 Ctxt.ReturnFileFromFolder()} method for proper tuning, e.g. to change the default file name or disable the HTTP 304 answers. In all cases, the file content will be served by the @88@ directly from the kernel mode, so will be very fast.
In order to have the BLOG content hosted in {\f1\fs20 root/blog} URI, you should specify the expected sub-URI when initializing your {\f1\fs20 TMVCApplication}:
!procedure TBlogApplication.Start(aServer: TSQLRestServer);
!begin
! ...
! fMainRunner := TMVCRunOnRestServer.Create(self,nil,'blog').
! ...
Here, any request to {\f1\fs20 blog.project.com} will be redirected to {\f1\fs20 root/blog}, so will match the expected {\f1\fs20 TBlogApplication} URIs. Note that by default, {\f1\fs20 TMVCRunOnRestServer.RunOnRestServerSub} will redirect any {\f1\fs20 root/blog} request to {\f1\fs20 root/blog/default}, so this URI will be transparent for the user.
: MVCViewModel
:  Defining the commands
The {\f1\fs20 MVCViewModel.pas} unit defines the {\i Controller} (or {\i ViewModel}) of the "{\i 30 - MVC Server}" sample application.\line It uses the {\f1\fs20 mORMotMVC.pas} unit , which is the main @*MVC@ kernel for the framework, allowing to easily create {\i Controllers} binding the ORM/SOA features ({\f1\fs20 mORMot.pas}) to the {\i @*Mustache@} Views ({\f1\fs20 SynMustache.pas}).
First of all, we defined an {\f1\fs20 interface}, with the expected methods corresponding to the various {\i commands} of the @*web application@:
!!  IBlogApplication = interface(IMVCApplication)
!!    procedure ArticleView(
!      ID: integer; var WithComments: boolean; Direction: integer;
!      out Article: TSQLArticle; out Author: TSQLAuthor;
!      out Comments: TObjectList);
!!    procedure AuthorView(
!      var ID: integer; out Author: variant; out Articles: variant);
!!    function Login(
!      const LogonName,PlainPassword: RawUTF8): TMVCAction;
!!    function Logout: TMVCAction;
!!    procedure ArticleEdit(
!      var ID: integer; const Title,Content: RawUTF8;
!      const ValidationError: variant;
!      out Article: TSQLArticle);
!!    function ArticleCommit(
!      ID: integer; const Title,Content: RawUTF8): TMVCAction;
!  end;
In fact, {\f1\fs20 IMVCApplication} is defined as such in {\f1\fs20 mORMotMVC.pas}:
!  IMVCApplication = interface(IInvokable)
!    ['{C48718BF-861B-448A-B593-8012DB51E15D}']
!    procedure Default(var Scope: variant);
!    procedure Error(var Msg: RawUTF8; var Scope: variant);
!  end;
As such, the {\f1\fs20 IBlogApplication} will define the following web pages, corresponding to each of its methods: {\i Default, Error, ArticleView, AuthorView, Login, Logout, ArticleEdit} and {\i ArticleCommit}. Each command of this application will map an URI, e.g. {\f1\fs20 /blog/default} or {\f1\fs20 /blog/login} - remember that our model defined {\f1\fs20 'blog'} as its root URI. You may let all commands be accessible from a sub-URI (e.g. {\f1\fs20 /blog/web/default}), but here this is not needed, since we are creating a "pure web" application.
Each command will have its own {\i View}. For instance, you will find {\f1\fs20 Default.html}, {\f1\fs20 Error.html} or {\f1\fs20 ArticleView.html} in the {\i "Views"} sub-folder of the sample. If you did not supply any file in this folder, some void files will be created.
Incoming method parameters of each method (i.e. defined as {\f1\fs20 const} or {\f1\fs20 var}) will be transmitted on the URI, encoded as regular HTTP parameters, whereas outgoing method parameters (i.e. defined as {\f1\fs20 var} or {\f1\fs20 out}) will be transmitted to the {\i View}, as data context for the rendering. Simple types are transmitted (like {\f1\fs20 integer} or {\f1\fs20 RawUTF8}); but you will also find ORM classes (like {\f1\fs20 TSQLAuthor}), an outgoing {\f1\fs20 TObjectList}, or some {\f1\fs20 variant} - which may be either values or a complex @80@.
In fact, you may find out that the {\i Login, Logout} and {\i ArticleCommit} methods do not have any outgoing parameters, but were defined as {\f1\fs20 function} returning a {\f1\fs20 TMVCAction} record.\line This type is declared as such in {\f1\fs20 mORMotMVC.pas}:
!  TMVCAction = record
!    RedirectToMethodName: RawUTF8;
!    RedirectToMethodParameters: RawUTF8;
!    ReturnedStatus: cardinal;
!  end;
Any method returning a {\f1\fs20 TMVCAction} content won't render directly any view, but will allow to go directly to another method, for proper rendering, just by providing a method name and some optional parameters.\line Note that even the regular views, i.e. the methods which do not have this {\f1\fs20 TMVCAction} parameter, may break the default rendering process on any error, raising an {\f1\fs20 EMVCApplication} exception which will in fact redirect the view to another page, mainly the {\f1\fs20 Error} page.
To better understand how it works, run the "{\i 30 - MVC Server}" sample. Remember that to be able to register the port #8092 for the {\f1\fs20 http.sys} server, you will need to run the {\f1\fs20 MVCServer.exe} program at least once with {\i Windows Administrator} rights - see @109@. Then point your browser to @http://localhost:8092/ - you will see the main page of the BLOG, filled with some random data. Quite some "blabla", to be sincere!
What you see is the {\f1\fs20 Default} page rendered. The {\f1\fs20 IBlogApplication.Default()} method has been called, then the outgoing {\f1\fs20 Scope} data has been rendered by the {\f1\fs20 Default.html} {\i Mustache} template.
If you click on an article title, it will go to @http://localhost:8092/blog/articleView?id=99 - i.e. calling {\f1\fs20 IBlogApplication.ArticleView()} with the {\f1\fs20 ID} parameter containing 99, and other incoming parameters (i.e. {\f1\fs20 WithComments} and {\f1\fs20 Direction}) set to their default value (i.e. respectively {\f1\fs20 false} and {\f1\fs20 0}). The {\f1\fs20 ArticleView()} method will then read the {\f1\fs20 TSQLArticle} data from the ORM, then send it to the {\f1\fs20 ArticleView.html} {\i Mustache} template.
Now, just change in your browser the URI from @http://localhost:8092/blog/articleView?id=99 (here we clicked on the {\f1\fs20 Article} with ID=99) into @http://localhost:8092/blog/articleView/json?id=99 (i.e. entering {\f1\fs20 /articleView{\b /json}} instead of {\f1\fs20 /articleView}, as a fake sub-URI).\line Now the browser is showing you the JSON data context, as transmitted to the {\f1\fs20 ArticleView.html} template. Just check both the JSON content and the corresponding {\i Mustache} template: I think you will find out how it works. Take a look at @81@ as reference.
From any blog article view, click on the "{\f1\fs20 Show Comments}" button: you are redirected to a new page, at URI @http://localhost:8092/blog/ArticleView?id=99&withComments=true#comments and now the comments corresponding to the article are displayed. If you click on the "{\f1\fs20 Previous}" or "{\f1\fs20 Next}" buttons, a new URI @http://localhost:8092/blog/ArticleView?id=99&withComments=true&direction=1 will be submitted: in fact, {\f1\fs20 direction=1} will search for the previous article, and we still have the {\f1\fs20 withComments=true} parameter set, so that the user will be able to see the comments, as expected. If you click on the "{\f1\fs20 Hide Comments}" button, the URI will change to be without any {\f1\fs20 withComments=true} parameter - i.e. @http://localhost:8092/blog/ArticleView?id=98#comments : now the comments won't be displayed.
The sequence is rendered as such:
\graph mORMotMVCSequence mORMot MVC/MVVM URI - Commands sequence
\/blog/default URI\Default()\routing + decode¤incoming params¤to controller method
\Default()\defaultjson\outgoing params¤encoded as JSON
\defaultjson\default.html¤template\rendering¤data context
\default.html¤template\main blog web page¤with list of articles\Mustache¤engine
\/blog/articleView?id=99\ArticleView(ID=99)\routing + decode¤incoming params¤to controller method
\ArticleView(ID=99)\articlejson\outgoing params¤encoded as JSON
\articlejson\articleView.html¤template\rendering¤data context
\articleView.html¤template\web page¤with one article¤without comments\Mustache¤engine
\/blog/articleView?id=99&withcomments=true\ArticleView(ID=99,WithComments=true)\routing + decode¤incoming params¤to controller method
\ArticleView(ID=99,WithComments=true)\articlecomment\outgoing params¤encoded as JSON
\articlecomment\ articleView.html¤template\rendering¤data context
\ articleView.html¤template\web page¤with one article¤and its comments\Mustache¤engine
=defaultjson={Scope:{articles:[....
=articlejson={WithComments=false,¤Article={ID=99,Author:1,...¤Comments:[],...
=articlecomment={WithComments=true,¤Article={ID=99,Author:1,...¤Comments:[¤{ID:163],...
\
In this diagram, we can see that each HTTP request is stateless, uncoupled from the previous. The user experience is created by changing the URI with additional parameters (like {\f1\fs20 withComments=true}).\line This is how the web works.
Then try to go to @http://localhost:8092/blog/mvc-info - and check out the page which appears.\line You will get all the information corresponding to your application, especially a list of all available commands:
$/blog/Default?Scope=..[variant]..
$/blog/Error?Msg=..[string]..&Scope=..[variant]..
$/blog/ArticleView?ID=..[integer]..&WithComments=..[boolean]..&Direction=..[integer]..
$/blog/AuthorView?ID=..[integer]..
$/blog/Login?LogonName=..[string]..&PlainPassword=..[string]..
$/blog/Logout
$/blog/ArticleEdit?ID=..[integer]..&Title=..[string]..&Content=..[string]..&ValidationError=..[variant]..
$/blog/ArticleCommit?ID=..[integer]..&Title=..[string]..&Content=..[string]..
And every view, including its data context, e.g.
$/blog/AuthorView?ID=..[integer]..
${{Main}}: variant
${{ID}}: integer
${{Author}}: TSQLAuthor
${{Articles}}: variant
You may use this page as reference when writing your {\i Mustache} Views. It will reflect the exact state of the running application.
:  Implementing the Controller
To build the application {\i Controller}, we will need to implement our {\f1\fs20 IBlogApplication interface}.
!  TBlogApplication = class(TMVCApplication,IBlogApplication)
!  ...
!  public
!    procedure Start(aServer: TSQLRestServer); reintroduce;
!    procedure Default(var Scope: variant);
!    procedure ArticleView(ID: integer; var WithComments: boolean;
!      Direction: integer;
!      out Article: TSQLArticle; out Author: variant;
!      out Comments: TObjectList);
!    ...
!  end;
We defined a new class, inheriting from {\f1\fs20 TMVCApplication} - as defined in {\f1\fs20 mORMotMVC.pas}, and implementing our expected interface. {\f1\fs20 TMVCApplication} will do all the low-level plumbing for you, using a set of implementation classes.
Let's implement a simple command:
!procedure TBlogApplication.AuthorView(var ID: integer; out Author: TSQLAuthor;
!  out Articles: variant);
!begin
!  RestModel.Retrieve(ID,Author);
!  if Author.ID<>0 then
!    Articles := RestModel.RetrieveListJSON(
!      TSQLArticle,'Author=? order by id desc limit 50',[ID],ARTICLE_FIELDS) else
!    raise EMVCApplication.CreateGotoError(HTTP_NOTFOUND);
!end;
By convention, all parameters are allocated when {\f1\fs20 TMVCApplication} will execute a method. So you do not need to allocate or handle the {\f1\fs20 Author: TSQLAuthor} instance lifetime.\line You have direct access to the underlying {\f1\fs20 TSQLRest} instance via {\f1\fs20 TMVCApplication.RestModel}: so all CRUD operations are available. You can let the ORM do the low level SQL work for you: to retrieve all information about one {\f1\fs20 TSQLAuthor} and get the list of its associated articles, we just use a {\f1\fs20 TSQLRest} method with the appropriate WHERE clause. Here we returned the list of articles as a {\f1\fs20 TDocVariant}, so that they will be transmitted as a JSON array, without any intermediate marshalling to {\f1\fs20 TSQLArticle} instances, but with the {\f1\fs20 Tags} dynamic array published property returned as an array of integers (you may have used {\f1\fs20 TObjectList} or {\f1\fs20 RawJSON} instead, as will be detailed below).\line In case of any error, an {\f1\fs20 EMVCApplication} will be raised: when such an exception happens, the {\f1\fs20 TMVCApplication} will handle and convert it into a page change, and a redirection to the {\f1\fs20 IBlogApplication.Error()} method, which will return an error page, using the {\f1\fs20 Error.html} view template.
Let's take a look at a bit more complex method, which we talked about in @%%mORMotMVCSequence@:
!procedure TBlogApplication.ArticleView(
!  ID: integer; var WithComments: boolean; Direction: integer;
!  out Article: TSQLArticle; out Author: variant; out Comments: TObjectList);
!var newID: TID;
!const WHERE: array[1..2] of PUTF8Char = (
!  'ID<? order by id desc','ID>? order by id');
!begin
!  if Direction in [1,2] then // allows fast paging using index on ID
!    if RestModel.OneFieldValue(TSQLArticle,'ID',WHERE[Direction],[],[ID],newID) and
!      (newID<>0) then
!      ID := newID;
!  RestModel.Retrieve(ID,Article);
!  if Article.ID<>0 then begin
!    Author := RestModel.RetrieveDocVariant(
!      TSQLAuthor,'ID=?',[Article.Author.ID],'FirstName,FamilyName');
!    if WithComments then begin
!      Comments.Free; // we will override the TObjectList created at input
!      Comments := RestModel.RetrieveList(TSQLComment,'Article=?',[Article.ID]);
!    end;
!  end else
!    raise EMVCApplication.CreateGotoError(HTTP_NOTFOUND);
!end;
This method has to manage several use cases:
- Display an {\f1\fs20 Article} from the database;
- Retrieve the {\f1\fs20 Author} first name and family name;
- Optionally display the associated {\f1\fs20 Comment}s;
- Optionally get the previous or next {\f1\fs20 Article};
- Trigger an error in case of an invalid request.
Reading the above code is enough to understand how those 5 features are implemented in this method. The incoming parameters, as triggered by the {\i Views}, are used to identify the action to be taken. Then {\f1\fs20 TMVCApplication.RestModel} methods are used to retrieve the needed information directly from the ORM. Outgoing parameters ({\f1\fs20 Article,Author,Comments}) are transmitted to the {\i Mustache} {\i View}, for rendering.
In fact, there are several ways to retrieve your data, using the {\f1\fs20 RestModel} ORM methods. For instance, in the above code, we used a {\f1\fs20 TObjectList} to transmit our comments.\line But we may have used a @80@ parameter:
!procedure TBlogApplication.ArticleView(
!  ID: integer; var WithComments: boolean; Direction: integer;
!  out Article: TSQLArticle; out Author: variant; out Comments: variant);
! ...
!    if WithComments then
!      Comments := RestModel.RetrieveDocVariantArray(TSQLComment,'','Article=?',[Article.ID],'');
In this case, data will be returned {\i per representation}, as {\f1\fs20 variant} values. Any {\i @*dynamic array@} properties will be identified in the {\f1\fs20 TSQLRecord}, and converted as proper array of values.
Or with a {\f1\fs20 RawJSON} kind of output parameter:
!procedure TBlogApplication.ArticleView(
!  ID: integer; var WithComments: boolean; Direction: integer;
!  out Article: TSQLArticle; out Author: variant; out Comments: RawJSON);
! ...
!    if WithComments then
!      Comments := RestModel.RetrieveListJSON(TSQLComment,'Article=?',[Article.ID],'');
Using a {\f1\fs20 RawJSON} will be in fact the fastest way of processing the information on the server side. But it will return the data directly from the database - as a consequence, dynamic arrays will be returned as a Base64-encoded blob.
It is up to you to choose the method and encoding needed for your exact context.\line If your purpose is just to retrieve some data and push it back to the view, {\f1\fs20 RawJSON} is fast, but a {\f1\fs20 TDocVariant} will also convert dynamic arrays to a proper JSON array. If you want to process the returned information with some business code, returning a {\f1\fs20 TObjectList} may be convenient if you need to run some {\f1\fs20 TSQLRecord} methods on the returned list.
 or a {\f1\fs20 TDocVariant} array may have its needs, if you want to create some meta-object gathering all information, e.g. for {\f1\fs20 Scope} as returned by the {\f1\fs20 Default} method:
!procedure TBlogApplication.Default(var Scope: variant);
! ...
!  if not fDefaultData.AddExistingProp('Archives',Scope) then
!    fDefaultData.AddNewProp('Archives',RestModel.RetrieveDocVariantArray(
!      TSQLArticle,'','group by PublishedMonth order by PublishedMonth desc limit 12',[],
!      'distinct(PublishedMonth),max(ID)+1 as FirstID'),Scope);
!end;
You can notice how the calendar months are retrieved from the database, using a safe {\f1\fs20 fDefaultData: ILockedDocVariant} private field to store the value as cache, in a thread-safe manner (we will see later more about how to implement thread-safety). If the {\f1\fs20 'Archives'} value is in the {\f1\fs20 fDefaultData} cache, it will be returned immediately as part of the {\f1\fs20 Scope} returned document. Otherwise, it will use {\f1\fs20 RestModel.RetrieveDocVariantArray} to retrieve the last 12 available months. When a new {\f1\fs20 Article} is created, or modified, {\f1\fs20 TBlogApplication.FlushAnyCache} will call {\f1\fs20 fDefaultData.Clear} to ensure that the updated information will be retrieved from the database on next {\f1\fs20 Default()} call.
The above ORM request will generate the following SQL statement:
$ SELECT distinct(PublishedMonth),max(ID)+1 as FirstID FROM Article
$  group by PublishedMonth order by PublishedMonth desc limit 12
The {\f1\fs20 Default()} method will therefore return the following JSON context:
${
$  "Scope": {
$    ...
$    "Archives":
$    [
$      {
$        "PublishedMonth": 24178,
$        "FirstID": 101
$      },
$      {
$        "PublishedMonth": 24177,
$        "FirstID": 100
$      },
$      ...
... which will be processed by the {\i Mustache} engine.\line If you put a breakpoint at the end of this {\f1\fs20 Default()} method, and inspect the "{\f1\fs20 Scope}" variable, the Delphi debugger will in fact show you in real time the exact JSON content, retrieved from the ORM.
I suspect you just find out how {\i mORMot}'s ORM/SOA abilites, and JSON / {\f1\fs20 TDocVariant} offer amazing means of processing your data. You have the best of both worlds: ORM/SOA gives you fixed structures and strong typing (like in C++/C#/Java), whereas {\f1\fs20 TDocVariant} gives you a flexible object scheme, using late-binding to access its content (like in Python/Ruby/JavaScript).
:  Variable input parameters
If you want to support a variable number of named parameters, you can define a {\f1\fs20 variant} input parameter, and provide the input as a JSON document, using a {\f1\fs20 TDocVariant} storage. But marshalling the context as JSON will involve using some JavaScript in the HTML page, which may not be very convenient.
If you want to handle a non-fixed set of regular URI or POST value, you can prefix all the incoming parameter names with the dotted name of a single defined {\f1\fs20 variant}.\line For instance, if you have the following controller method:
! function TnWebMVCMenu.CadastroSalvar3(const p: variant): TMVCAction;
Then you can supply as parameter at URI level:
$ p.a1=5&p.a2=dfasdfa
And you will be able to handle them in the controller body:
!function TnWebMVCMenu.CadastroSalvar3(const p: variant): TMVCAction;
!begin
!  GotoView(result,'Cadastro',
!    ['pp1',p.a1,
!     'pp2',p.a2])
!end;
You are now free to specify some versatile HTML forms in your views, and provide the controller with any kind of input parameters.\line Of course, it may sound safer and easier to explicitly define and name each one of the input parameters, with simple types like {\f1\fs20 integer} or {\f1\fs20 RawUTF8}. But this convention may help you work with any kind of HTML views.
:  Using Services in the Controller
Any controller method could retrieve and execute any dependency from its {\f1\fs20 interface}, following the {\i @*IoC@} pattern - see @157@.\line You have two ways of performing the dependency resolution:
- From the associated {\f1\fs20 TSQLRest.Services} container;
- From its own protected {\f1\fs20 Resolve()} method, since {\f1\fs20 TMVCApplication} inherits from {\f1\fs20 TInjectableObject}.
In fact, you can set up your {\f1\fs20 TMVCApplication} instance to use any external dependencies, including stubs and mocks, or high-level DDD services (e.g. repository or modelization process), using its {\f1\fs20 CreateInjected()} constructor instead of plain {\f1\fs20 Create}.
:111  Controller Thread Safety
When run from a {\f1\fs20 TSQLRestServer} instance, our {\i MVC} application commands will be executed by default without any thread protection. When hosted within a {\f1\fs20 TSQLHttpServer} instance - see @88@ - several threads may execute the same {\i Controller} methods at the same time. It is therefore up to your application code to ensure that your {\f1\fs20 TMVCApplication} process is thread safe.
Note that by design, all {\f1\fs20 TMVCApplication.RestModel} ORM methods are thread-safe.\line If your {\i Controller} business code only uses ORM methods, sending back the information to the {\i Views}, without storing any data locally, it will be perfectly thread safe.\line See for instance the {\f1\fs20 TBlogApplication.AuthorView} method we described above.
But consider this method (simplified from the real "{\i 30 - MVC Server}" sample):
!type
!  TBlogApplication = class(TMVCApplication,IBlogApplication)
!  protected
!    fDefaultArticles: variant;
!   ...
!
!procedure TBlogApplication.Default(var Scope: variant);
!begin
!  if VarIsEmpty(fDefaultArticles) then
!    fDefaultArticles := RestModel.RetrieveDocVariantArray(
!      TSQLArticle,'','order by ID desc limit 20',[],ARTICLE_FIELDS);
!  _ObjAddProps(['Articles',fDefaultArticles],Scope);
!end;
In fact, even if this method may sound safe, we have an issue when it is executed by several threads: one thread may be assigning a value to {\f1\fs20 fDefaultArticles}, whereas another thread may be using the {\f1\fs20 fDefaultArticles} content. This may result into random runtime errors, very difficult to solve. Even creating a local variable may not be safe, since any access to {\f1\fs20 fDefaultArticles} should be protected.
A first - and brutal - solution could be to force the {\f1\fs20 TSQLRestServer} instance to execute all method-based services (including our {\i MVC} commands) in a giant lock, as stated about @25@:
! aServer.AcquireExecutionMode[execSOAByMethod] := amLocked; // or amBackgroundThread
But this may slow down the whole server process, and reduce its scaling abilities.
You could also lock explicitly the {\i Controller} method, for instance:
!procedure TBlogApplication.Default(var Scope: variant);
!begin
!!  Locker.ProtectMethod;
!  if VarIsEmpty(fDefaultData) then
! ...
Using the {\f1\fs20 TMVCApplication.Locker: IAutoLocker} is a simple and efficient way of protecting your method. In fact, {\f1\fs20 @*TAutoLocker@} class' {\f1\fs20 ProtectMethod} will return an {\f1\fs20 IUnknown} variable, which will let the compiler create an hidden {\f1\fs20 try .. finally} block in the method body, to release the lock when it quits. But this locker will be shared by the whole {\f1\fs20 TMVCApplication} instance, so will be like a giant lock on your {\i Controller} process.
A more tuned and safe implementation may be to use a {\f1\fs20 ILockedDocVariant} instead of a plain {\f1\fs20 TDocVariant} for caching the data. You may therefore write:
!type
!  TBlogApplication = class(TMVCApplication,IBlogApplication)
!  protected
!    fDefaultData: ILockedDocVariant;
!   ...
!procedure TBlogApplication.Start(aServer: TSQLRestServer);
!begin
!  fDefaultData := TLockedDocVariant.Create;
!  ...
!
!procedure TBlogApplication.Default(var Scope: variant);
!begin
!  if not fDefaultData.AddExistingProp('Articles',Scope) then
!    fDefaultData.AddNewProp('Articles',RestModel.RetrieveDocVariantArray(
!      TSQLArticle,'','order by ID desc limit 20',[],ARTICLE_FIELDS),Scope);
!end;
Using {\f1\fs20 ILockedDocVariant} will ensure that only access to this resource will be locked (no giant lock any more), and that slow ORM process (like {\f1\fs20 RestModel.RetrieveDocVariantArray}) will take place lock-free, to maximize the resource usage.\line This is in fact the pattern used by the "{\i 30 - MVC Server}" sample. Even @63@ may benefit from this {\f1\fs20 @**TLockedDocVariant@} kind of storage, for efficient multi-thread process - see @72@.
:  Web Sessions
@*Sessions@ are usually implemented via cookies, in web sites. A login/logout procedure enhances security of the @*web application@, and User experience can be tuned via small persistence of client-driven data. The {\f1\fs20 TMVCApplication} class allows creating such sessions.
You can store whatever information you need within the client-side cookie. {\f1\fs20 TMVCSessionWithCookies} allows to define a {\f1\fs20 record}, which will be used to store the information as optimized binary, in the browser cache. You can use this cookie information as a cache to the current session, e.g. storing the logged user display name, his/her preferences or rights - avoiding a round trip to the database.\line Of course, you should never trust the cookie content (even if our format uses secure encryption, and a digital signature via a {\f1\fs20 HMAC-CRC32C} algorithm). But you can use it as a convenient cache, always checking the real data in the database when you are about to perform any security-related action. The cookie also stores an integer Session ID, and issuing and expiration dates: as such, it matches all {\f1\fs20 @*JWT@} ({\i Javascript Web Token}) - see @http://jwt.io - features, as signature, encryption, and {\i jwi/iat/exp} claims, with a smaller overhead, and without using unsafe Web Local Storage.
For our "{\i 30 - MVC Server}" sample application, we defined the following {\f1\fs20 record} in {\f1\fs20 MVCViewModel.pas}:
!  TCookieData = packed record
!    AuthorName: RawUTF8;
!    AuthorID: cardinal;
!    AuthorRights: TSQLAuthorRights;
!  end;
This record will be serialized in two ways:
- As raw binary, without the field names, within the cookie, after Base64 encoding of encrypted and digitally signed data;
- As a JSON object, with explicit field names, when transmitted to the {\i Views} as {\f1\fs20 "Session"} data context.
In order to have proper JSON serialization of the {\f1\fs20 record}, you will need to specify its structure, if you use a version of Delphi without the new RTII (i.e. before Delphi 2010) - see @51@.
Then we can use the {\f1\fs20 TMVCApplication.CurrentSession} property to perform the authentication, after successful login:
!function TBlogApplication.Login(const LogonName, PlainPassword: RawUTF8): TMVCAction;
!var Author: TSQLAuthor;
!!    SessionInfo: TCookieData;
!begin
!!  if CurrentSession.CheckAndRetrieve<>0 then begin
!    GotoError(result,HTTP_BADREQUEST);
!    exit;
!  end;
!!  Author := TSQLAuthor.Create(RestModel,'LogonName=?',[LogonName]);
!  try
!!    if (Author.ID<>0) and Author.CheckPlainPassword(PlainPassword) then begin
!      SessionInfo.AuthorName := Author.LogonName;
!      SessionInfo.AuthorID := Author.ID;
!      SessionInfo.AuthorRights := Author.Rights;
!!      CurrentSession.Initialize(@SessionInfo,TypeInfo(TCookieData));
!!      GotoDefault(result);
!    end else
!      GotoError(result,sErrorInvalidLogin);
!  finally
!    Author.Free;
!  end;
!end;
As you can see, this {\f1\fs20 Login()} method will be triggered from @http://localhost:8092/blog/login with {\f1\fs20 LogonName=...&plainpassword=...} parameters. It will first check that there is no current session, retrieve the ORM {\f1\fs20 Author} corresponding to the {\f1\fs20 LogonName}, check the supplied password, and set the {\f1\fs20 SessionInfo: TCookieData} structure with the needed information.\line A call to {\f1\fs20 CurrentSession.Initialize()} will compute the cookie, then prepare to send it to the client browser.
The {\f1\fs20 Login()} method returns a {\f1\fs20 TMVCAction} structure. As a consequence, the call to {\f1\fs20 GotoDefault(result)} will let the {\f1\fs20 TMVCApplication} processor render the {\f1\fs20 Default()} method, as if the {\f1\fs20 /blog/default} URI will have been requested. On invalid credential, an error page is displayed instead.
When a web page is computed, the following overridden method will be executed:
!function TBlogApplication.GetViewInfo(MethodIndex: integer): variant;
!begin
!  result := inherited GetViewInfo(MethodIndex);
!!  _ObjAddProps(['blog',fBlogMainInfo,
!!    'session',CurrentSession.CheckAndRetrieveInfo(TypeInfo(TCookieData))],result);
!end;
It will append the session information from the cookie to the returned {\i View} data context, as such:
${
$  "Scope": {
$    "articles":
$    ...
$  },
$  "main": {
$    "pageName": "Default",
$    "blog": {
$      "Title": "mORMot BLOG",
$      ...
$    },
$!    "session": {
!$      "AuthorName": "synopse",
!$      "AuthorID": 1,
!$      "AuthorRights": {
!$        "Comment": true,
!$        "Post": true,
!$        "Delete": true,
!$        "Administrate": true
!$      },
$      "id": 1
$    }
$  }
$}
Here, the {\f1\fs20 session} object will contain the {\f1\fs20 TCookieData} information, ready to be processed by the {\i Mustache View} - e.g. as {\f1\fs20 session.AuthorName}. In addition, your view may include some buttons for logged-only features, like comments or content edition, using {\f1\fs20 boolean} fields defined in {\f1\fs20 session.AuthorRights}.
For security reasons, before actually performing an action requiring a specific right, it is preferred to check from the Model if the user is effectively allowed. An attacker may have forged a fake cookie - even if it is very unlikely, since cookies are encrypted and signed. It is a good approach to treat all cookies information as an unsafe cache, acceptable for most operation, but which should always be dual-checked.\line So your server code will call {\f1\fs20 CurrentSession.CheckAndRetrieve} then access the data {\f1\fs20 RestModel} for verification before any sensitive action is performed. Defining a common method could be handy:
!function TBlogApplication.GetLoggedAuthorID(Right: TSQLAuthorRight;
!  ContentToFillAuthor: TSQLContent): TID;
!var SessionInfo: TCookieData;
!    author: TSQLAuthor;
!begin
!  result := 0;
!!  if (CurrentSession.CheckAndRetrieve(@SessionInfo,TypeInfo(TCookieData))>0) and
!!     (Right in SessionInfo.AuthorRights) then
!!    with TSQLAuthor.AutoFree(author,RestModel,SessionInfo.AuthorID) do
!!    if Right in author.Rights then begin
!      result := SessionInfo.AuthorID;
!      if ContentToFillAuthor<>nil then begin
!        ContentToFillAuthor.Author := pointer(result);
!        ContentToFillAuthor.AuthorName := author.LogonName;
!      end;
!    end;
!end;
It will be used as such, e.g. to verify if a user can comment an article:
!function TBlogApplication.ArticleComment(ID: TID;
!  const Title,Comment: RawUTF8): TMVCAction;
!var comm: TSQLComment;
!    AuthorID: TID;
!    error: string;
!begin
!  with TSQLComment.AutoFree(comm) do begin
!!    AuthorID := GetLoggedAuthorID(canComment,comm);
!    if AuthorID=0 then begin
!      GotoError(result,sErrorNeedValidAuthorSession);
!      exit;
!    end;
!  ...
Eventually, when the browser asks for the {\f1\fs20 /blog/logout} URI, the following method will be executed:
!function TBlogApplication.Logout: TMVCAction;
!begin
!!  CurrentSession.Finalize;
!  GotoDefault(result);
!end;
The session cookie will then be deleted on the browser side.
Note that if any deprecated or invalid cookie is detected by the {\i mORMot} MVC server, it will also be automatically deleted on the browser side.
: Writing the Views
See @81@ for a description of how rendering take place in this MVC/MVVM application. You will find the @*Mustache@ templates in the "{\f1\fs20 Views}" sub-folder of the "{\i 30 - MVC Server}" sample application.
You will find some {\f1\fs20 *.html} files, one per command expecting a {\f1\fs20 View}, and some {\f1\fs20 *.partial} files, which are some kind of re-usable sub-templates - we use them to easily compute the page header and footer, and to have a convenient way of gathering some piece of template code, to be re-used in several {\f1\fs20 *.html} views.
Here is how {\f1\fs20 Default.html} is defined:
$$!{{>header}}
$$!{{>masthead}}
$$      <div class="blog-header">
$$!        <h1 class="blog-title">{{main.blog.title}}</h1>
$$!        <p class="lead blog-description">{{main.blog.description}}</p>
$$      </div>
$$      <div class="row">
$$        <div class="col-sm-8 blog-main">
$$!{{#Scope}}
$$!{{>articlerow}}
$$!        {{#lastID}}
$$!        <p><a href="default?scope={{.}}" class="btn btn-primary btn-sm">Previous Articles</a></p>
$$!        {{/lastID}}
$$        </div>
$$        <div class="col-sm-3 col-sm-offset-1 blog-sidebar">
$$          <div class="sidebar-module sidebar-module-inset">
$$            <h4>About</h4>
$$!            {{{WikiToHtml main.blog.about}}}
$$          </div>
$$          <div class="sidebar-module">
$$            <h4>Archives</h4>
$$            <ol class="list-unstyled">
$$!              {{#Archives}}
$$!              <li><a href="default?scope={{FirstID}}">{{MonthToText PublishedMonth}}</a></li>
$$!              {{/Archives}}
$$            </ol>
$$          </div>
$$  </div>
$$   </div>
$$!{{/Scope}}
$$!{{>footer}}
The {\f1\fs20 \{\{>partials\}\}} are easily identified, as other {\f1\fs20 \{\{...\}\}} value tags. The main partial is {\f1\fs20 \{\{>articlerow\}\}}, which will display all articles list.\line {\f1\fs20 \{\{\{WikiToHtml main.blog.about\}\}\}} is an {\i Expression Block} able to render some simple text into proper HTML, using a simple Wiki syntax.\line {\f1\fs20 \{\{MonthToText PublishedMonth\}\}} will execute a custom {\i Expression Block}, defined in our {\f1\fs20 TBlogApplication}, which will convert the obfuscated {\f1\fs20 TSQLArticle.PublishedMonth} integer value into the corresponding name and year:
!procedure TBlogApplication.MonthToText(const Value: variant;
!  out result: variant);
!const MONTHS: array[0..11] of string = (
!  'January','February','March','April','May','June','July','August',
!  'September','October','November','December');
!var month: integer;
!begin
!  if VariantToInteger(Value,month) and (month>0) then
!    result := MONTHS[month mod 12]+' '+IntToStr(month div 12);
!end;
The page displaying the {\f1\fs20 Author} information is in fact quite simple:
$${{>header}}
$${{>masthead}}
$$      <div class="blog-header">
$$        <h1 class="blog-title">User {{Author.LogonName}}</h1>
$$  <div class="lead blog-description">{{Author.FirstName}} {{Author.FamilyName}}
$$  </div>
$$      </div>
$$   <div class="panel panel-default">
$$   <div class="panel-heading">Information about <strong>{{Author.LogonName}}</strong></div>
$$   <div class="panel-body">
$$!      {{{TSQLAuthor.HtmlTable Author}}}
$$   </div>
$$   </div>
$$!{{>articlerow}}
$${{>footer}}
It will share the same {\f1\fs20 \{\{>partials\}\}}, for a consistent and maintainable web site design, but in fact most of the process will take place by the magic of two tags:
- {\f1\fs20 \{\{\{TSQLAuthor.HtmlTable Author\}\}\}} is an {\i Expression Block} linked to {\f1\fs20 TMVCApplication.RestModel} @*ORM@, which will create a HTML table - with the syntax expected by our {\i @*BootStrap@} {\f1\fs20 CSS} - for a {\f1\fs20 TSQLAuthor} record, identifying the property types and display them as expected (e.g. for dates or time stamps, or for enumerates or sets).
- {\f1\fs20 \{\{>articlerow\}\}} is a partial also shared with {\f1\fs20 ArticleView.html}, which will render a list of {\f1\fs20 TSQLArticle} encoded as {\f1\fs20 \{\{#Articles\}\}}...{\f1\fs20 \{\{/Articles\}\}} sections.
Take a look at the {\f1\fs20 mORMotMVC.pas} unit: you will discover that every aspect of the MVC process has been divided into small classes, so that the framework is able to create web applications, but also any kind of MVC applications, including mobile or VCL/FMX apps, and/or reporting - using {\f1\fs20 mORMotReport.pas}.
:75Hosting
%cartoon02.png
We could identify several implementation patterns of a {\i mORMot} server and its clients:
- Stand-alone application, either in the same process or locally on the same computer;
- Private self-hosting, e.g. in a corporate network, with a {\i mORMot} executable or service publishing some content to clients locally or over the Internet (directly from a DMZ or via a VPN);
- @**Cloud@ hosting, using a dedicated server in a data-center, or any cloud solution based on virtualization;
- Mixed hosting, using @*CDN@ network services to cache most of the requests of your {\i mORMot} server.
As we already stated, our @35@ allow all these patterns.\line We will now detail some hosting schemes.
\page
:112 Windows and Linux hosted
The current version of the framework fully supports deploying the {\i mORMot} servers on the {\i @**Windows@} platform, either as a {\i Win32} executable, or - for latest versions of the {\i Delphi} compiler - as a {\i Win64} executable.
{\i @**Linux@} support (via @**FPC@ 3.2.x) is available, but we face some FPC compiler-level issue with FPC 2.x, which does not supply the needed {\f1\fs20 interface} RTTI - see @http://bugs.freepascal.org/view.php?id=26774 - so that the SOA and MVC features are not working directly non old FPC revision, so you need to generate the RTTI from a Delphi compiler, as stated @125@. For the client side, there is no limitation, thanks to our @86@, which is perfectly supported even by oldest FPC compiler under {\i Linux}. The {\i Linux} backend available in latest Delphi is not supported, since FPC 2.x gives pretty good results (we use it on production since years), and a Delphi Entreprise licence is required to access it - which we don't have.
In practice, a {\i mORMot} server expects much lower hardware requirements (in CPU, storage and RAM terms) than a regular {\f1\fs20 IIS-WCF-MSSQL-.Net} stack. And it requires almost no maintenance.
As a consequence, the potential implementation schemes could be hosted as such:
- Stand-alone application, without any explicit server;
- Self-hosted service running on the corporate file server, or on a small dedicated VM or recycled computer (for best performance, just put your data on a new SSD on the old hardware PC);
- @*Cloud@ services running {\i Windows Server}, with minimal configuration: {\f1\fs20 IIS}, {\f1\fs20 .Net} or {\f1\fs20 MS SQL} are not necessary at all - a cheap virtual system with 512 MB of memory is enough to run your {\i mORMot} service and serve hundredths of clients;
- {\i Linux} servers, with no dependency (even latest version of {\i SQlite3} is statically linked to the executables), using less hardware resource.
In the cloud, since every resource used is monitored and billed, you would like to minimize RAM use: you should better take a look at @http://www.delphitools.info/2013/11/20/moving-hosts-now-settled and @http://www.delphitools.info/2013/11/29/flush-windows-file-cache for practical advices and feedbacks.
About the edition of {\i Windows} to be used, of course IT people will ensure you that {\i Windows Server} is mandatory. But from our tests, you will obtain pretty good results, even with a regular Windows 7 or 8 version of the operating system. On the other side, it is not serious to envisage hosting a server on Windows XP, which is not supported any more by Microsoft - even if technically a {\i mORMot} server will work very well on this deprecated platform.
Of course, if you use @27@, the hardware and hosting expectations may vary. It will depend on the database back-end used, and will necessarily be much more demanding than our internal {\i SQLite3} database engine. In practice, a {\i mORMot} server using a {\i SQLite3} engine running on a SSD hardware, in {\f1\fs20 lmExclusive} mode - see @60@ - runs faster than most @*SQL@ or @*NoSQL@ engines available, since it will be hosted within the {\i mORMot} server process itself - see @4@.
: Deployment Architecture
About @**hosting@ architecture, the easiest is to have your main {\f1\fs20 TSQLRestServer} class handling the service, in conjunction with other Client-Server process (like ORM). See @%%mORMotDesign1@ about this generic Client-Server architecture and the "Shared server" next paragraph.
But you may find out some (good?) reasons which main induce another design:
- For better scalability, you should want to use a dedicated process (or even dedicated hardware) to split the database and the service process;
- For @*security@ reasons, you want to expose only services to your Internet clients, and setup a @*DMZ@ hosting only services, and separate database with logic instance;
- Services are not the main part of your business, and you would like to enable or disable the published SOA scope, on demand;
- To implement an efficient solution for the most complex kind of application, as provided by @54@;
- Your main data will be hosted on high performance SSD / NAS drives with safe RAID, but some data should better be hosted on cheaper storage (e.g. @85@);
- You are selling one product, to be run on several environments (debugging / production, starter / corporate editions, centralized / P2P design...), depending on your clients demand;
- Whatever your IT people or managers want {\i mORMot} to.
Also consider per-table @*redirection@ - see @93@, or @147@, for even more advanced hosting abilities. See for instance the @%%HostingRedirection@ and @%%HostingReplication@ diagrams.
The possibilities are endless, so we will here below only present some typical use-cases.
:  Shared server
This is the easiest configuration: one HTTP server instance, which serves both ORM and Services. On practice, this is perfectly working and scalable.
\graph mORMotServices1 Service Hosting on mORMot - shared server
\Internet (VPN)\Local Network
node [shape=box];
\Local Network\HTTP server
subgraph cluster_0 {
"Client 1\n(Delphi)";
label="PC 1";
}
subgraph cluster_1 {
"Client 2\n(AJAX)";
label="PC 2";
}
subgraph cluster_2 {
\HTTP server\ORM
\HTTP server\Service
label="PC Server";
}
\Client 1¤(Delphi)\Local Network\JSON + REST¤over HTTP/1.1
\Client 2¤(AJAX)\Internet (VPN)\JSON + REST¤over HTTP/1.1
\
You can tune this solution, as such:
- Setting the group user rights properly - see @18@ - you can disable the remote ORM access from the Internet, for the AJAX Clients - but allow rich {\i Delphi} clients (like {\f1\fs20 PC1}) to access the ORM;
- You can have direct in-process access to the service interfaces from the ORM, and vice-versa: if your services and @*ORM@ are deeply inter-dependent, direct access will be the faster solution.
\page
:  Two servers
In this configuration, two physical servers are available:
- A network DMZ is opened to serve only service content over the Internet, via "HTTP server 2";
- Then on the local network, "HTTP server 1" is used by both PC 1 and Services to access the ORM;
- Both "PC Client 1" and the ORM core are able to connect to Services via a dedicated "HTTP server 3".
\graph mORMotServices2 Service Hosting on mORMot - two servers
"Internet (VPN)";
"Local Network";
node [shape=box];
\Local Network\HTTP server 1
\Internet (VPN)\HTTP server 2
\Services\HTTP server 1
subgraph cluster_0 {
"Client 1\n(Delphi)";
label="PC 1";
}
subgraph cluster_1 {
"Client 2\n(AJAX)";
label="PC 2";
}
subgraph cluster_2 {
\HTTP server 2\Services
\HTTP server 3\Services
label="PC Server DMZ";
}
subgraph cluster_3 {
\HTTP server 1\ORM
\ORM\HTTP server 3
label="PC Server internal";
}
\Local Network\HTTP server 3
\Client 1¤(Delphi)\Local Network\JSON + REST¤over HTTP/1.1
\Client 2¤(AJAX)\Internet (VPN)\JSON + REST¤over HTTP/1.1
\
Of course, the database will be located on "PC Server internal", i.e. the one hosting the ORM, and the Services will be one regular client: so we may use @39@ on purpose to enhance performance. In order to access the remote ORM features, and provide a communication endpoint to the embedded services, a {\f1\fs20 @*TSQLRestServerRemoteDB@} kind of server class can be used.
\page
:  Two instances on the same server
This is the most complex configuration. In this case, only one physical server is deployed:
- A dedicated "HTTP server 2" instance will serve service content over the Internet (via a DMZ configuration of the associated network card);
- "PC Client 1" will access to the ORM via "HTTP server 1", or to services via "HTTP server 3";
- For performance reasons, since ORM and Services are on the same computer, using named pipes (or even local Windows Messages) instead of slower HTTP-TCP/IP is a good idea: in such case, ORM will access services via "Named Pipe server 2", whereas Services will serve their content to the ORM via "Named Pipe server 1".
\graph mORMotServices3 Service Hosting on mORMot - one server, two instances
"Internet (VPN)";
"Local Network";
node [shape=box];
\Local Network\HTTP server 1
\Local Network\HTTP server 3
\Internet (VPN)\HTTP server 2
\Services\Named Pipe server 1
subgraph cluster_0 {
"Client 1\n(Delphi)";
label="PC 1";
}
subgraph cluster_1 {
"Client 2\n(AJAX)";
label="PC 2";
}
subgraph cluster_2 {
\Named Pipe server 1\ORM
\Named Pipe server 2\Services
\ORM\Named Pipe server 2
\HTTP server 2\Services
\HTTP server 3\Services
\HTTP server 1\ORM
label="PC Server";
}
\Client 1¤(Delphi)\Local Network\JSON + REST¤over HTTP/1.1
\Client 2¤(AJAX)\Internet (VPN)\JSON + REST¤over HTTP/1.1
\
Of course, you can make any combination of the protocols and servers, to tune hosting for a particular purpose. You can even create several ORM servers or Services servers (grouped per features family or per product), which will cooperate for better scaling and performance.
If you consider implementing a @*stand-alone@ application for hosting your services, and has therefore basic ORM needs (e.g. you may need only CRUD statements for handling authentication), you may use the lighter {\f1\fs20 TSQLRestServerFullMemory} kind of server instead of a full {\f1\fs20 TSQLRestServerDB}, which will embed a {\i @*SQLite3@} database engine, perhaps not worth it in this case.
:97  Scaling via CDN
Our beloved stateless @9@ model, in conjunction with @96@ will enable several levels of caching, from a local proxy cache - see e.g. @http://www.squid-cache.org or @http://www.varnish-cache.org - or an external {\i Content Delivery Network} (@**CDN@) service - e.g. @http://www.cloudflare.com
Your {\i mORMot} server may be able to publish some dynamic HTML pages, or simple generic JSON services, and then let the CDN do the caching. An expiration time out of 30 seconds, configured at CDN level, will definitively help your @*web application@ to scale to thousands of visitors.
\graph mORMotServices4 Service Hosting on mORMot - Content Delivery Network (CDB)
\Local Network A\Internet
\Local Network B\Internet
\3G/4G Network\Internet
node [shape=box];
subgraph cluster_0 {
\Client A1¤(Delphi)\Local Network A
\Client A2¤(Delphi)\Local Network A
\Client A3¤(AJAX)\Local Network A
label="Office A";
}
subgraph cluster_1 {
\Client B1¤(Delphi)\Local Network B
\Client B2¤(Delphi)\Local Network B
label="Office B";
}
subgraph cluster_2 {
\Client C¤(Mobile AJAX)\3G/4G Network
label="Mobile";
}
\Internet\CDN US
\Internet\CDN UK
subgraph cluster_3 {
\CDN US\Cache
label="  US";
}
subgraph cluster_4 {
\CDN UK\ Cache
label="       UK";
}
subgraph cluster_5 {
\CDN UK\mORMot¤Server
\CDN US\mORMot¤Server
\mORMot¤Server\Database
label=" Data Center";
}
\Internet\mORMot¤Server
\
In practice, static content - see @95@ - or some simple JSON requests - returned via {\f1\fs20 Ctxt.Results()} or an interface-based service - will benefit of using such a CDN.
When any client requests the {\i mORMot} server URI, it will be in fact redirected to the closest CDN node available. For instance, some client in Canada will be redirected to the "CDN US" server, or one mobile client in France will be redirected to the "CDN UK" server.
Then each CDN will check if the requested URI is already in its cache, according to its settings, and the expiration parameters which may be set within the HTTP headers of the cache header. If the resource is in local cache, it will be returned to the client immediately. If the resource is not in its cache, the CDN node will ask the {\i mORMot} server, cache the returned content, then return this content to the client. Any further attempt to this URI, compatible with the expiration parameters, won't trigger any request to the {\i mORMot} server.
Of course, you can define some URI patterns to never be cached, and point directly to the {\i mORMot} server. All authenticated services, for instance will need direct access to the {\i mORMot} server, since the @98@ will append a session-private signature to each URI. Just ensure that you disabled authentication - using {\f1\fs20 TSQLRestServer.ServiceMethodByPassAuthentication()} for method-based services, or {\f1\fs20 TServiceFactoryServer.ByPassAuthentication} property for interface-based services. The per-session signature appended at each URI will indeed void any attempt of third-party cache.
If your project starts to have success, using a CDN is an easy and cheap way of increasing your number of clients. Your {\i mORMot} server will focus on its own purpose, which may be safe storage, authentication and high-level SOA, then let the remaining content be served by such a third-party caching system.
:43Security
%cartoon03.png
The framework tries to implement @**security@ via:
- Process safety;
- Authentication;
- Authorization.
{\i Process safety} is implemented at every @7@ level:
- Strong @*encryption@ to keep information private - see @151@ and @187@;
- @*Atomic@ity of the {\i @*SQLite3@} database - see @60@;
- @15@ architecture to avoid most synchronization issues;
- @13@ associated to the Object pascal @*strong type@ syntax;
- Extended test coverage - see @12@ - of the framework core.
{\i Authentication} allows user identification:
- Build-in optional @*authentication@ mechanism, implementing both {\i per-user @*session@s} and individual REST {\i @*Query Authentication@};
- {\i Authentication groups} are used for proper authorization;
- Several authentication schemes, from very secure @*HMAC-SHA256@ based challenging to weak but simple authentication;
- Class-based architecture, allowing custom extension.
{\i Authorization} of a given process is based on the group policy, after proper authentication:
- {\i Per-table access right} functionalities built-in at lowest level of the framework;
- Per-method execution policy for interface-based services;
- General high-level security attributes, for SQL or Service remote execution.
Process safety has already been documented (see links above).
We will now give general information about both authentication and authorization in the framework.
\page
:18 Authentication
Extracted from {\i Wikipedia}:
{\i Authentication (from Greek: "real" or "genuine", from "author") is the act of confirming the truth of an attribute of a datum or entity. This might involve confirming the identity of a person or software program, tracing the origins of an artifact, or ensuring that a product is what its packaging and labeling claims to be. Authentication often involves verifying the validity of at least one form of identification.}
:  Principles
How to handle @*authentication@ in a @*REST@ful @*Client-Server@ architecture is a matter of debate.
Commonly, it can be achieved, in the @*SOA@ over @*HTTP@ world via:
- HTTP {\i basic auth} over @*HTTPS@;
- {\i Cookies} and @*session@ management;
- {\i @*Query Authentication@} with additional signature parameters.
We'll have to adapt, or even better mix those techniques, to match our framework architecture at best.
Each authentication scheme has its own PROs and CONs, depending on the purpose of your security policy and software architecture:
|%30%24%24%22
|\b Criteria|HTTPS {\i basic auth}|Cookies+Session|Query Auth.\b0
|Browser integration|Native|Native|Via {\i @*JavaScript@}
|User Interaction|Rude|Custom|Custom
|Web Service use\line (rough estimation)|95%|4%|1%
|Session handling|Yes|Yes|No
|Session managed by|Client|Server|N/A
|Password on Server|Yes|Yes/No|N/A
|Truly Stateless|Yes|No|Yes
|Truly RESTful|No|No|Yes
|HTTP-free|No|No|Yes
|%
:   HTTP basic auth over HTTPS
This first solution, based on the standard @**HTTPS@ protocol, is used by most web services. It's easy to implement, available by default on all browsers, but has some known draw-backs, like the awful authentication window displayed on the Browser, which will persist (there is no {\i LogOut}-like feature here), some server-side additional CPU consumption, and the fact that the user-name and password are transmitted (over HTTPS) into the Server (it should be more secure to let the password stay only on the client side, during keyboard entry, and be stored as secure hash on the Server).
The supplied {\f1\fs20 TSQLHttpClientWinHTTP} and {\f1\fs20 TSQLHttpClientWinINet} clients classes are able to connect using HTTPS, and the {\f1\fs20 THttpApiServer} server class can send compatible content.
:   Session via Cookies
To be honest, a @**session@ managed on the Server is not truly @*Stateless@. One possibility could be to maintain all data within the cookie content. And, by design, the cookie is handled on the Server side (Client in fact don’t even try to interpret this cookie data: it just hands it back to the server on each successive request). But this cookie data is application state data, so the client should manage it, not the server, in a pure Stateless world.
The cookie technique itself is HTTP-linked, so it's not truly RESTful, which should be protocol-independent. Since our framework does not provide only HTTP protocol, but offers other ways of transmission, Cookies were left at the baker's home.
:   Query Authentication
{\i @**Query Authentication@} consists in signing each RESTful request via some additional parameters on the URI. See @http://broadcast.oreilly.com/2009/12/principles-for-standardized-rest-authentication.html about this technique. It was defined as such in this article:
{\i All REST queries must be authenticated by signing the query parameters sorted in lower-case, alphabetical order using the private credential as the signing token. Signing should occur before URI encoding the query string.}
For instance, here is a generic URI sample from the link above:
$ GET /object?apiKey=Qwerty2010
should be transmitted as such:
$ GET /object?timestamp=1261496500&apiKey=Qwerty2010&signature=abcdef0123456789
The string being signed is "{\f1\fs20 /object?apikey=Qwerty2010&timestamp=1261496500}" and the signature is the {\i @*SHA256@} hash of that string using the private component of the API key.
This technique is perhaps the more compatible with a Stateless architecture, and can also been implemented with a light @*session@ management.
Server-side data caching is always available. In our framework, we cache the responses at the SQL level, not at the URI level (thanks to our optimized implementation of {\f1\fs20 GetJSONObjectAsSQL}, the URI to SQL conversion is very fast). So adding this extra parameter doesn't break the cache mechanism.
:  Framework authentication
Even if, theoretically speaking, {\i @*Query Authentication@} sounds to be the better for implementing a truly @*REST@ful architecture, our framework tries to implement a @*Client-Server@ design.
In practice, we may consider two way of using it:
- With no authentication nor user right management (e.g. for local access of data, or framework use over a secured network);
- With per-user authentication and right management via defined {\i security groups}, and a per-query authentication, following several protocols (a set of {\i mORMot} flavors, Windows NTLM/Kerberos, or any custom scheme).
According to RESTful principle, handling per-@*session@ data is not to be implemented in such  architecture. A minimal "session-like" feature was introduced only to handle user authentication with very low overhead on both Client and Server side. The main technique used for our security is therefore {\i Query Authentication}, i.e. a per-URI signature.
If the {\f1\fs20 aHandleUserAuthentication} parameter is left to its default {\f1\fs20 false} value for the {\f1\fs20 TSQLRestServer. Create constructor}, no authentication is performed. All tables will be accessible by any client, as stated in @19@. As stated above, for security reasons, the ability to execute {\f1\fs20 INSERT / UPDATE / DELETE} SQL statement via a RESTful {\i POST} command is never allowed by default with remote connections: only {\f1\fs20 SELECT} can be executed via this {\i POST} verb.
If authentication is enabled for the Client-Server process (i.e. if the {\f1\fs20 aHandleUserAuthentication} parameter is set to {\f1\fs20 true} at the {\f1\fs20 TSQLRestServer} instance construction), the following security features will be added:
- On the Server side, a dedicated service, accessible via the {\f1\fs20 ModelRoot/Auth} URI is to be called to register an User, and create an in-memory session;
- Client {\i should} open a session to access to the Server, and after authentication validation (e.g. via {\f1\fs20 UserName / Password} pair, or Windows credentials);
- Each @*CRUD@ statement is checked against the authenticated User security group, via the {\f1\fs20 AccessRights} column and its {\f1\fs20 GET / POST / PUT / DELETE} per-table bit sets;
- Thanks to {\i Per-User} authentication, any SQL statement commands may be available via the RESTful {\i POST} verb for an user with its {\f1\fs20 AccessRights} group field containing a {\f1\fs20 reSQL} flag in its {\f1\fs20 AllowRemoteExecute};
- Each REST request will expect an additional parameter, named {\f1\fs20 session_signature}, to every URL. Using the URI instead of {\i cookies} allows the signature process to work with all communication protocols, not only @*HTTP@;
- Of course, you have the opportunity to tune or even by-pass the security for a given service (method-based or interface-based), on need: e.g. to allow some methods only to your system administrators, or to serve public HTML content.
:139   Per-User authentication
On the Server side, two tables, defined by the {\f1\fs20 @**TSQLAuthGroup@} and {\f1\fs20 @**TSQLAuthUser@} classes, will handle respectively per-group access rights (authorization), and user validation (authentication). In the database, they will be persisted as {\f1\fs20 AuthGroup} and {\f1\fs20 AuthUser} tables.
The Server will search for any class inheriting from {\f1\fs20 TSQLAuthGroup} and {\f1\fs20 TSQLAuthUser} in its @*Model@. By default, you can use plain {\f1\fs20 TSQLAuthGroup} and {\f1\fs20 TSQLAuthUser} classes - and if none is defined in the model, and authentication is enabled, those mandatory classes will be added. But you can inherit from {\f1\fs20 TSQLAuthGroup} and {\f1\fs20 TSQLAuthUser}, and define e.g. your own fields, for any custom purpose at {\i Group} or {\i User} level. The exact class types are available from {\f1\fs20 SQLAuthUserClass} and {\f1\fs20 SQLAuthGroupClass} properties of {\f1\fs20 TSQLRestServer}.
Since the whole records will be loaded and persisted in memory at every authentication, do not store too much data in those tables: for instance, do not put historical data (like previous client activity), or huge BLOBs (like detailed pictures) - a dedicated table or set of tables will be a better idea.
Here is the layout of the default {\f1\fs20 AuthGroup} table, as defined by the {\f1\fs20 TSQLAuthGroup} class type:
\graph DBAuthGroup AuthGroup Record Layout
rankdir=LR;
node [shape=Mrecord];
struct1 [label="ID : TID|AccessRights : RawUTF8|Ident : RawUTF8|SessionTimeout : integer"];
\
The {\f1\fs20 AccessRights} column is a textual CSV serialization of the {\f1\fs20 TSQLAccessRights} record content, as expected by the {\f1\fs20 TSQLRestServer.URI} method. Using a CSV serialization, instead of a binary serialization, will allow the change of the {\f1\fs20 MAX_SQLTABLES} constant value.
The {\f1\fs20 AuthUser} table, as defined by the {\f1\fs20 TSQLAuthUser} class type, is defined as such:
\graph DBAuthUser AuthUser Record Layout
rankdir=LR;
node [shape=Mrecord];
struct1 [label="ID : TID|Data : TSQLRawBlob|DisplayName : RawUTF8|<f0>GroupRights : TSQLAuthGroup|LogonName : RawUTF8|PasswordHashHexa : RawUTF8"];
struct2 [label="AuthGroup"];
struct1:f0 -> struct2;
\
Each user has therefore its own associated {\f1\fs20 AuthGroup} table, a name to be entered at login, a name to be displayed on screen or reports, and a @*SHA256@ hash of its registered password (with optional {\f1\fs20 @*PBKDF2_HMAC_SHA256@} derivation). A custom {\f1\fs20 Data} BLOB field is specified for your own application use, but not accessed by the framework.
By default, the following security groups are created on a void database:
|%14%12%14%11%11%12%12%12
|\b Group|POST SQL|SELECT SQL|Auth R|Auth W|Tables R|Tables W|Services\b0
|Admin|Yes|Yes|Yes|Yes|Yes|Yes|Yes
|Supervisor|No|Yes|Yes|No|Yes|Yes|Yes
|User|No|No|No|No|Yes|Yes|Yes
|Guest|No|No|No|No|Yes|No|No
|%
'{\i Admin}' will be the only able to execute remote not SELECT SQL statements for POST commands ({\f1\fs20 reSQL} flag in {\f1\fs20 TSQLAccessRights. AllowRemoteExecute}) and modify the {\f1\fs20 Auth*} tables (i.e. {\f1\fs20 AuthUser} and {\f1\fs20 AuthGroup}).\line  '{\i Admin}' and '{\i Supervisor}' will allow any SELECT SQL statements to be executed, even if the table can't be retrieved and checked (corresponding to the {\f1\fs20 reSQLSelectWithoutTable} flag).\line '{\i User}' won't have the {\f1\fs20 reSQLSelectWithoutTable} flag, nor the right to retrieve the {\f1\fs20 Auth*} tables data for other users.\line '{\i Guest}' won't have access to the interface-based remote JSON-RPC service (no {\f1\fs20 reService} flag), nor perform any modification to a table: in short, this is an ORM read-only limited user.
Please see @19@ and the {\f1\fs20 TSQLAccessRights} documentation for all available options and use cases.
Then the corresponding '{\i Admin}', '{\i Supervisor}' and '{\i User}' {\f1\fs20 AuthUser} accounts are created, with the default '{\i synopse}' password.
{\b You MUST override those default '{\i synopse}' passwords for each {\f1\fs20 AuthUser} record to a custom genuine value.}
A typical JSON representation of the default security user/group definitions are the following:
$[{"AuthUser": [
${"RowID":1, "LogonName":"Admin", "DisplayName":"Admin", "PasswordHashHexa":"67aeea294e1cb515236fd7829c55ec820ef888e8e221814d24d83b3dc4d825dd", "GroupRights":1, "Data":null},
${"RowID":2, "LogonName":"Supervisor", "DisplayName":"Supervisor", "PasswordHashHexa":"67aeea294e1cb515236fd7829c55ec820ef888e8e221814d24d83b3dc4d825dd", "GroupRights":2, "Data":null},
${"RowID":3, "LogonName":"User", "DisplayName":"User", "PasswordHashHexa":"67aeea294e1cb515236fd7829c55ec820ef888e8e221814d24d83b3dc4d825dd", "GroupRights":3, "Data":null}]},
${"AuthGroup": [
${"RowID":1, "Ident":"Admin", "SessionTimeout":10, "AccessRights":"11,1-256,0,1-256,0,1-256,0,1-256,0"},
${"RowID":2, "Ident":"Supervisor", "SessionTimeout":60, "AccessRights":"10,1-256,0,3-256,0,3-256,0,3-256,0"},
${"RowID":3, "Ident":"User", "SessionTimeout":60, "AccessRights":"10,3-256,0,3-256,0,3-256,0,3-256,0"},
${"RowID":4, "Ident":"Guest", "SessionTimeout":60, "AccessRights":"0,3-256,0,0,0,0"}]}]
Of course, you can change {\f1\fs20 AuthUser} and {\f1\fs20 AuthGroup} table content, to match your security requirements, and application specifications. You can specify a per-table @*CRUD@ access, via the {\f1\fs20 AccessRights} column, as we stated above, speaking about the {\f1\fs20 TSQLAccessRights} record layout.
This will implement both {\i Query Authentication} together with a group-defined {\i per-user right} management.
:   Session handling
A dedicated RESTful service, available from the {\f1\fs20 ModelRoot/Auth} URI, is to be used for user authentication, handling so called @**session@s.
In {\i mORMot}, a very light in-memory set of sessions is implemented:
- The unique {\f1\fs20 ModelRoot/Auth} URI end-point will create a session after proper authorization;
- {\i In-memory} session allows very fast process and reactivity, on Server side;
- Sessions could be optionally persisted on disk at server shutdown, to avoid breaking existing client connections;
- An {\f1\fs20 integer} {\i session identifier} is used for all authorization process, independently from the underlying authentication scheme (i.e. {\i mORMot} is not tied to cookies, and its session process is much more generic).
Those sessions are in-memory {\f1\fs20 TAuthSession} class instances. Note that this class does not inherit from a {\f1\fs20 TSQLRecord} table so won't be remotely accessible, for performance and security reasons.
The server methods should not have to access those {\f1\fs20 TAuthSession} instances directly, but rely on the {\f1\fs20 SessionID} identifier. But you can still access the current session properties, e.g. the remote user, thanks to methods like {\f1\fs20 TSQLRestServer.SessionGetUser(): TSQLAuthUser}.
When the Client is about to close (typically in {\f1\fs20 TSQLRestClientURI.Destroy}), a {\f1\fs20 GET ModelRoot/auth?UserName=...&Session=...} request is sent to the remote server, in order to explicitly close the corresponding session in the server memory (avoiding most {\i re-play} attacks).
Note that each opened session has an internal {\i TimeOut} parameter (retrieved from the associated {\f1\fs20 TSQLAuthGroup} table content): after some time of inactivity, sessions are closed on the Server Side.
In addition, sessions are used to manage safe cross-client @**transaction@s:
- When a transaction is initiated by a client, it will store the corresponding client Session ID, and use it to allow client-safe writing;
- Any further write to the DB (Add/Update/Delete) will be accessible only from this Session ID, until the transaction is released (via commit or rollback);
- If a transaction began and another client session try to write on the DB, it will wait until the current transaction is released - a timeout may occur if the server is not able to acquire the write status within some time;
- This global write locking is defined by the {\f1\fs20 TSQLRest.AcquireWriteMode / AcquireWriteTimeOut} properties, and used on the Server-Side by {\f1\fs20 TSQLRestServer.URI} - you can change this behavior by setting e.g. {\f1\fs20 AcquireWriteMode := amBackgroundThread} which will lock any write process to be executed in a dedicated thread: this may be mandatory is your database client expects the transaction process to take place in the same thread (e.g. @*MS SQL@);
- If the server do not handle Session/Authentication, transactions can be unsafe, in a multi-client concurrent architecture.
For performance reasons in a multi-client environment, it's mandatory to release a transaction (via commit or rollback) as soon as possible, using e.g. @28@, or - even better - write dedicated @63@ which will process the whole transaction in one step, following the @100@.
You can specify an optional file name parameter to the {\f1\fs20 TSQLRestServer.Shutdown()} method, which will save the current server state into a local file. Then, if you restart the server in a short time, you may be able to restore all session information by using {\f1\fs20 TSQLRestServer.SessionsLoadFromFile()}. This feature will enable e.g. a quick and transparent ORM Server executable upgrade, in production. But note that even if sessions are persisted and able to be restored, any session-dependent complex data - like server-side temporary @92@ as generated by interface-based services - won't be available. This session temporary backup/restore will make sense only when the server is in @*ORM@ mode, {\i not} when used as @*SOA@.
:  Authentication schemes
:   Class-driven authentication
Authentication is implemented in {\i mORMot} via the following classes:
\graph HierTSQLRestServerAuthentication TSQLRestServerAuthentication classes hierarchy
\TSQLRestServerAuthenticationSSPI\TSQLRestServerAuthenticationSignedURI
\TSQLRestServerAuthenticationDefault\TSQLRestServerAuthenticationSignedURI
\TSQLRestServerAuthenticationSignedURI\TSQLRestServerAuthenticationURI
\TSQLRestServerAuthenticationNone\TSQLRestServerAuthenticationURI
\TSQLRestServerAuthenticationURI\TSQLRestServerAuthentication
\TSQLRestServerAuthenticationHttpAbstract\TSQLRestServerAuthentication
\TSQLRestServerAuthenticationHttpBasic\TSQLRestServerAuthenticationHttpAbstract
\
In fact, you can use one of the following @*REST@ful authentication schemes:
|%50%50
|\b class|Scheme\b0
|{\f1\fs20 TSQLRestServerAuthenticationDefault}|{\i mORMot} secure authentication, as a proprietary dual-pass challenge and @*SHA256@/@*PBKDF2_HMAC_SHA256@ hashing
|{\f1\fs20 TSQLRestServerAuthenticationSSPI}|@*Windows authentication@, via the logged user
|{\f1\fs20 TSQLRestServerAuthenticationNone}|Weak but simple authentication, based on user name
|{\f1\fs20 TSQLRestServerAuthenticationHttpBasic}|HTTP Basic authentication\line Warning: password is not encrypted
|%
All those classes will identify a {\f1\fs20 TSQLAuthUser} record from a user name. The associated {\f1\fs20 TSQLAuthGroup} is then used later for authorization.
You can add you own custom authentication scheme by defining your own class, inheriting from {\f1\fs20 TSQLRestServerAuthentication}.
By default, no authentication is performed.
If you set the {\f1\fs20 aHandleUserAuthentication} parameter to {\f1\fs20 true} when calling the constructor {\f1\fs20 TSQLRestServer.Create()}, both default secure {\i mORMot} authentication and Windows authentication are available. In fact, the {\f1\fs20 constructor} executes the following:
!constructor TSQLRestServer.Create(aModel: TSQLModel; aHandleUserAuthentication: boolean);
!  (...)
!  if aHandleUserAuthentication then
!    // default mORMot authentication schemes
!!    AuthenticationRegister([
!!      TSQLRestServerAuthenticationDefault,TSQLRestServerAuthenticationSSPI]);
!  (...)
In order to define one or several authentication scheme, you can call the {\f1\fs20 AuthenticationRegister()} and {\f1\fs20 AuthenticationUnregister()} methods of {\f1\fs20 TSQLRestServer}.
:98   mORMot secure RESTful authentication
The {\f1\fs20 TSQLRestServerAuthenticationDefault} class implements a proprietary but secure RESTful @18@.
As stated in @74@, typical client-side authentication is performed using the following command:
! MyClient.SetUser('User','synopse'); // default user for Security tests
Here are the typical steps to be followed in order to create a new user session via {\i mORMot} authentication scheme:
- Client sends a {\f1\fs20 GET ModelRoot/auth?UserName=...} request to the remote server - with the above command, it will be {\f1\fs20 GET ModelRoot/auth?UserName=User};
- Server answers with a hexadecimal {\i nonce} contents (valid for about 5 minutes), encoded as JSON result object;
- Client sends a {\f1\fs20 GET ModelRoot/auth?UserName=...&PassWord=...&ClientNonce=...} request to the remote server, in which {\f1\fs20 ClientNonce} is a random value used as Client {\i nonce}, and {\f1\fs20 PassWord} is computed from the log-on and password entered by the User, using both Server and Client {\i nonce} as salt;
- Server checks that the transmitted password is valid, i.e. that its matches the hashed password stored in its database and a time-valid Server {\i nonce} - if the value is not correct, authentication fails;
- On success, Server will create a new in-memory session and return the session number and a private key to be used during the session (encoded as JSON result object);
- On any further access to the Server, a {\f1\fs20 &session_signature=} parameter is added to the URL, and will be checked against the valid sessions in order to validate the request.
{\i @**Query Authentication@} is handled at the Client side in {\f1\fs20 TSQLRestClientURI.SessionSign} method, by computing the {\f1\fs20 session_signature} parameter for a given URL, according to the {\f1\fs20 TSQLRestServerAuthentication} class used.
In order to enhance security, the {\f1\fs20 session_signature} parameter will contain, encoded as 3 hexadecimal 32-bit cardinals:
- The @*Session@ ID (to retrieve the private key used for the signature);
- A Client Time Stamp (in 256 ms resolution) which must be greater or equal than the previous time stamp received;
- The URI signature, using the session private key, the user hashed password, and the supplied Client Time Stamp as source for its {\i @*crc32@} hashing algorithm.
Such a classical 3 points signature will avoid most {\i man-in-the-middle} (MITM) or {\i re-play} attacks.
Here is typical signature to access the {\f1\fs20 root} URL
$ root?session_signature=0000004C000F6BE365D8D454
In this case, {\f1\fs20 0000004C} is the Session ID, {\f1\fs20 000F6BE3} is the client time stamp (aka nonce), and {\f1\fs20 65D8D454} is the signature, computed by the following {\i Delphi} expression:
!(crc32(crc32(fPrivateSaltHash,PTimestamp,8),pointer(aURL),aURLlength)=aSignature);
For instance, a RESTful GET of the {\f1\fs20 TSQLRecordPeople} table with RowID=6 will have the following URI:
$ root/People/6?session_signature=0000004C000F6DD02E24541C
For better Server-side performance, the URI signature will use fast {\i crc32} hashing method, and not the more secure (but much slower) @*SHA256@. Since our security model is not officially validated as a standard method (there is no standard for per URI authentication of RESTful applications), the better security will be handled by encrypting the whole transmission channel, using standard @*HTTPS@ with certificates signed by a trusted CA, validated for both client and server side. The security involved by using {\i @*crc32@} will be enough for most common use. Note that the password hashing and the session opening will use @*SHA256@ or @*PBKDF2_HMAC_SHA256@, to enhance security with no performance penalty.
In our implementation, for better Server-side reaction, the {\f1\fs20 session_signature} parameter is appended at the end of the URI, and the URI parameters are not sorted alphabetically, as suggested by the reference article quoted above. This should not be a problem, either from a {\i Delphi} Client or from a @*AJAX@ / {\i JavaScript} client.
On practice, this scheme is secure and very fast, perfect for a {\i Delphi} client, or an AJAX application. If you expect a higher level of security for the URI signature, you may consider setting a cryptographic-level MD5/SHA1/SHA256/SHA512 hash, by selecting a given {\f1\fs20 TSQLRestServerAuthenticationSignedURIAlgo} on server side.
:121   Authentication using Windows credentials
:    Windows Authentication
By default, the {\i hash} of the user password is stored safely on the server side. This may be an issue for corporate applications, since a new user name / password pair is to be defined by each client, which may be annoying.
Since revision 1.18 of the framework, {\i mORMot} is able to use {\i @*Windows Authentication@} to identify any user. That is, the user does not need to enter any name nor password, but her/his Windows credentials, as entered at Windows session startup, will be used.
If the {\f1\fs20 SSPIAUTH} conditional is defined (which is the default), any call to {\f1\fs20 TSQLRestClientURI.SetUser()} method with a void {\f1\fs20 aUserName} parameter will try to use current logged name and password to perform a secure Client-Server authentication. It will in fact call the {\f1\fs20 class function TSQLRestServerAuthenticationSSPI.ClientSetUser()} method.
In this case, the {\f1\fs20 aPassword} parameter will identify if either {\i @**NTLM@} or {\i @**Kerberos@} authentication scheme is to be used: it may contain the @*SPN@ domain name to enabled {\i Kerberos} - see next section. This will be transparent to the framework, and a regular session will be created on success.
Only prerequisite is that the {\f1\fs20 TSQLAuthUser} table shall contain a corresponding entry, with its {\f1\fs20 LogonName} column equals to '{\i DomainName\UserName}' value. This data row won't be created automatically, since it is up to the application to allow or disallow access from an authenticated user: you can be member of the domain, but not eligible to the application.
:    Using NTLM or Kerberos
{\i Kerberos} is the preferred authentication protocol for Windows Server 2003 and subsequent Active Directory domains.
{\i Kerberos} authentication offers the following advantages over {\i NTLM} authentication:
- Mutual authentication.\line When a client uses the {\i Kerberos} protocol for authentication with a particular service on a particular server, {\i Kerberos} provides the client with an assurance that the service is not being impersonated by malicious code on the network.
- Simplified trust management.\line Networks with multiple domains no longer require a complex set of explicit, point-to-point trust relationships.
- Enhanced security.\line The old {\i NTLM} protocol suffers from several weaknesses, which have been fixed by {\i Kerberos}.
- Performance.\line Offers improved performance, mostly for server applications.
Requirements for {\i Kerberos} authentication are the following:
- Client and Server must join a domain, and the trusted third party must exist; if client and server are in different domain, these two domains must be configured as two-way trust.
- @**SPN@ must have been registered properly. {\i Service Principal Name} (SPNs) are unique identifiers for services running on servers. Each service that will use {\i Kerberos} authentication needs to have an SPN set for it so that clients can identify the service on the network. It is registered in {\i Active Directory} under either a computer account or a user account. See below for corresponding instructions.
Typical use case of either Kerberos or NTLM are defined by the {\f1\fs20 aPassword} parameter:
- {\i Kerberos} is used for a remote connection over a network and if {\f1\fs20 aPassword} is set to the expected SPN domain;
- {\i NTLM} is used over network connection if {\f1\fs20 aPassoword} is empty;
- {\i NTLM} is used when making local connection.
Note that {\i Kerberos} is used only when making remote connection over a network; {\i NTLM} is used when making local connection..
To enable {\i Kerberos} authentication in {\i mORMot}, you need to register SPN for your service.
The format of an SPN is {\f1\fs20 ServiceClass/Host:Port/ServiceName}. Typically, SPN for your service, developed with {\i mORMot}, looks like {\f1\fs20 mymormotservice/myserver.mydomain.tld} or {\f1\fs20 http/myserver.mydomain.tld}.
To list SPNs of a computer named {\f1\fs20 MYSERVER}, at the command prompt, type:
$    setspn -l myserver
Typically, you can see the following output:
$    Registered ServicePrincipalNames for CN=MYSERVER,OU=Computers,DC=domain,DC=tld:
$    HOST/MYSERVER.domain.tld
$    HOST/MYSERVER
If your service runs under {\f1\fs20 SYSTEM} or {\f1\fs20 Network Service} machine accounts, you can test {\i Kerberos} authentication by setting the {\f1\fs20 aPassword} parameter to value {\f1\fs20 'HOST/MYSERVER.domain.tld'} in the client code and run the application.
To register SPN for your service, at the command prompt, type:
- If your service run under {\f1\fs20 SYSTEM} or {\f1\fs20 Network Service} machine accounts:
$  setspn -a mymormotservice/myserver.mydomain.tld myserver
- If your service run under another domain account:
$   setspn -a mymormotservice/myserver.mydomain.tld myserviceaccount
Membership in {\i Domain Admins} group, or equivalent, is the minimum required to complete this procedure.
See @http://technet.microsoft.com/en-us/library/cc731241.aspx for more details.
After registration, you can connect to the server as such:
!  MyClient.SetUser('','mymormotservice/myserver.mydomain.tld'); // will use Kerberos
For good old NTLM, you can run:
!  MyClient.SetUser('',''); // will use NTLM
Or directly call the {\f1\fs20 TSQLRestServerAuthenticationSSPI.ClientSetUser()} method.
The authentication mode used will appear in the log file, if you define {\f1\fs20 WITHLOG} conditional when building the service application and if {\f1\fs20 sllUserAuth} is in {\f1\fs20 TSQLLog.Family.Level} set.
Messages will be as follows:
$ NTLM Authentication success for domain\myuser
$ Kerberos Authentication success for domain\myuser
The framework authorization will then be processed as usual, for all features like RESTful ORM process and remote services.
:   Weak authentication
The {\f1\fs20 TSQLRestServerAuthenticationNone} class can be used if you trust your client (e.g. via a {\f1\fs20 https} connection). It will implement a weak but simple authentication scheme.
Here are the typical steps to be followed in order to create a new user session via this authentication scheme:
- Client sends a {\f1\fs20 GET ModelRoot/auth?UserName=...} request to the remote server;
- Server checks that the transmitted user name is valid, i.e. that it is available in the {\f1\fs20 TSQLAuthGroup} table - if the value is not correct, authentication fails
- On success, Server will create a new in-memory session and returns the associated session number (encoded as decimal in the JSON result object);
- On any further access to the Server, a {\f1\fs20 &session_signature=} parameter is to be added to the URL with the correct session ID (encoded as hexadecimal), and will be checked against the valid sessions in order to validate the request.
For instance, a RESTful GET of the {\f1\fs20 TSQLRecordPeople} table with RowID=6 will have the following URI:
$ root/People/6?session_signature=0000004C
Here is some sample code about how to define this authentication scheme:
!  // on the Server side:
!  Server.AuthenticationRegister(TSQLRestServerAuthenticationNone);
!  ...
!  // on the Client side:
!  TSQLRestServerAuthenticationNone.ClientSetUser(Client,'User');
The performance benefit is very small in comparison to {\f1\fs20 TSQLRestServerAuthenticationDefault}, so should not be used for {\i Delphi} clients.
:   HTTP Basic authentication
The {\i Basic Authentication} mechanism provides no confidentiality protection for the transmitted credentials. They are merely encoded with {\i Base64} in transit, but not encrypted or hashed in any way. {\i Basic Authentication} is, therefore, typically used over HTTPS.
The {\f1\fs20 TSQLRestServerAuthenticationHttpBasic} class can be used to enable HTTP Basic authentication.\line This class is not to be used with a {\i mORMot} client, since {\f1\fs20 TSQLRestServerAuthenticationDefault} provides a much better scheme, both safer and faster, but could be used in conjunction with some browser clients, over HTTPS.
:  Clients authentication
:   Client interactivity
Note that with this design, it's up to the Client to react to an authentication error during any request, and ask again for the User pseudo and password at any time to create a new session. For multiple reasons (server restart, session timeout...) the session can be closed by the Server without previous notice.
In fact, the Client should just use create one instance of the {\f1\fs20 TSQLRestClientURI} classes as presented in @6@, then call the {\f1\fs20 SetUser} method as such:
!      Check(Client.SetUser('User','synopse')); // use default user
Then an event handled can be associated to the {\f1\fs20 TSQLRestClientURI. OnAuthentificationFailed} property, in order to ask the user to enter its login name and password:
!  TOnAuthentificationFailed = function(Retry: integer;
!    var aUserName, aPassword: string): boolean;
Of course, if {\i @*Windows Authentication@} is defined (see above), this event handler shall be adapted as expected. For instance, you may add a custom notification to register the corresponding user to the {\f1\fs20 TSQLAuthUser} table.
:   Authentication using AJAX
@90@ can generate {\i @*JavaScript@} code from its IDE. Our template-based code generation make this solution perfectly integrated with our {\i mORMot} server, especially about authentication: you will find the same {\f1\fs20 TSQLRestServerAuthenticationDefault} and {\f1\fs20 TSQLRestServerAuthenticationNone} classes in our {\f1\fs20 SynCrossPlatformREST.pas} unit, ready to authenticate to the server.\line In fact, there is also a command-line compiler available (named {\f1\fs20 smsc.exe}) which can create a {\f1\fs20 .js} file from {\i @*SmartPascal@} code: you may use it to integrate the generated client to a regular HTML5 application (using e.g. {\i JQuery} or {\i AngularJS}).
Some stand-alone working {\i JavaScript} code has been published in our forum by a framework user (thanks, "RangerX"), which implements the authentication schema as detailed above. It uses {\f1\fs20 jQuery}, and HTML 5 {\f1\fs20 LocalStorage}, not cookies, for storing session information on the Client side.\line See @https://synopse.info/forum/viewtopic.php?pid=2995#p2995
:198  JWT Authentication
As an alternative, you may use @192@ for authentication.
On server side, you can assign a {\f1\fs20 TJWTAbstract} inherited instance to {\f1\fs20 TSQLRestServer.JWTForUnauthenticatedRequest} so that any client providing a valid {\f1\fs20 @*JWT@} would be allowed to execute some requests.
!aJWTEngine := TJWTS3256.Create(aSecretKey, 60000, [jrcIssuer, jrcExpirationTime], [], aExpireMinutes);
!aRestServer.JWTForUnauthenticatedRequest := aJWTEngine; // will be owned by aRestServer
On mORMot client side, you can provide a valid {\f1\fs20 JWT} via:
!aRestClientURI.SessionHttpHeader := AuthorizationBearer(aJWT);
Any kind of JSON/HTTPS client could easily connect to such a service, by providing a valid {\f1\fs20 JWT} as {\f1\fs20 'Authorization: Bearer ####'} HTTP header. A dedicated authentication service may be used to return some {\f1\fs20 JWT} in exchange from some credential (typically a user-name/password) for your application.
In practice, for your internal {\i MicroServices} communication, you could therefore use regular @98@ over a @150@, which are pretty stable and efficient. Then for a public API, you could use a regular {\f1\fs20 TSQLHttpServer} - see @140@ - perhaps over a nginx reverse proxy (e.g. for {\i Let's Encrypt} HTTPS certification). Since the {\i mORMot} authentication is proprietary, using a {\f1\fs20 JWT} may sound more natural for a public API service, with a more relaxed JSON encoding and no contract. That is, on the server side, you define: {\f1\fs20 ServiceDefine(...).ResultAsJSONObjectWithoutResult := true;} and on the client side, you call the {\f1\fs20 TSQLRestClientURI.ServiceDefineSharedAPI()} method to follow a similar more standard JSON layout (i.e. JSON objects as input/output and not a JSON array without any contract negotiation).
By design, such clients won't be tied to any associated {\i mORMot} session or user. So @63@ should be only of {\f1\fs20 sicSingle}, {\f1\fs20 sicShared} or {\f1\fs20 sicPerThread} kind - see @92@.
See also {\f1\fs20 TSQLRestServer.JWTForUnauthenticatedRequestWhiteIP} for additional security, to require the client to connect from a finite set of allowed IP addresses.
\page
: Authorization
By @*authorization@, we mean the action to define an access policy to the @*REST@ful resources, for an authenticated user. Even if this user may be a a guest user (with no specific access credential), it should be identified as such, e.g. to serve public content.
The main principle is the {\i principle of least privilege} (also known as the {\i principle of minimal privilege} or the {\i principle of least authority}): in a particular abstraction layer of a computing environment, every module (such as a process, a user or a program depending on the subject) must be able to access only the information and resources that are necessary for its legitimate purpose.
It is most of the time implemented e.g. via {\i Access Control Lists} (ACL), set of capabilities or user groups. In {\i mORMot}, we defined user groups, associated to {\f1\fs20 TSQLAuthGroup} ORM class.
Today, authorization is part of a trust chain:
- In corporate networks, the {\i Active Directory} service gives a token for an already signed user, or LDAP allows access to resources;
- In social networks, protocols like {\i OAuth} allows to trust an user between services.
This allows the very convenient feature of {\i single sign-on}: the user can authenticate only once (e.g. at Windows logon), then he/she will be authenticated for its whole session, and each authorization will provide the appropriate rights. Our framework e.g. features @*NTLM@ / @*Kerberos@ authentication, as we just saw.
:  Per-table access rights
Even if authentication is disabled, a pointer to a {\f1\fs20 TSQLAccessRights} record, and its {\f1\fs20 GET / POST / PUT / DELETE} fields, is sent as a member of the parameter to the unique access point of the server class:
!procedure TSQLRestServer.URI(var Call: TSQLRestServerURIParams);
This will allow checking of access right for all @*CRUD@ operations, according to the table invoked. For instance, if the table {\f1\fs20 TSQLRecordPeople} has 2 as index in {\f1\fs20 TSQLModel.Tables[]}, any incoming {\f1\fs20 POST} command for {\f1\fs20 TSQLRecordPeople} will be allowed only if the 2nd bit in {\f1\fs20 RestAccessRights^.POST} field is set, as such:
!    case URI.Method of
!    mPOST: begin       // POST=ADD=INSERT
!      if URI.Table=nil then begin
!      (...)
!    end else
!      // here, Table<>nil and TableIndex in [0..MAX_SQLTABLES-1]
!!      if not (URI.TableIndex in Call.RestAccessRights^.POST) then // check User
!!        Call.OutStatus := HTTP_FORBIDDEN else
!      (...)
Making access rights a parameter allows this method to be handled as pure stateless, @*thread-safe@ and @*session@-free, from the bottom-most level of the framework.
On the other hand, the security policy defined by this global parameter does not allow tuned per-user authorization. In the current implementation, the {\f1\fs20 SUPERVISOR_ACCESS_RIGHTS} constant is transmitted for all handled communication protocols (direct access, Windows Messages, named pipe or HTTP). Only direct access via {\f1\fs20 @*TSQLRestClientDB@} will use {\f1\fs20 FULL_ACCESS_RIGHTS}, i.e. will have {\f1\fs20 AllowRemoteExecute} parameter set to all possible flags.
The light session process, as implemented by @18@, is used to override the access rights with the one defined in the {\f1\fs20 TSQLAuthGroup.AccessRights} field.
Be aware that this per-table access rights depend on the table order as defined in the associated {\f1\fs20 TSQLModel}. So if you add some tables to your database model, please take care to add the new tables {\i after} the existing. If you insert the new tables within current tables, you will need to update the access rights values.
:19  Additional safety
A {\f1\fs20 AllowRemoteExecute: TSQLAllowRemoteExecute} field has been made available in the {\f1\fs20 TSQLAccessRights} record to tune remote execution, depending on the authenticated user, and the group he/she is part of.
This field adds some flags to tune the security policy, for both @*SQL@ or @*SOA@ dimensions.
:   SQL remote execution
In our @*REST@ful implementation, the POST command with no table associated in the URI allows to execute any SQL statement directly. A GET command could also be used, either with the SQL statement transmitted as body (which is convenient, but not supported by all HTTP clients, since it is not standard), or inlined at URI level.
These special commands should be carefully tested before execution, since SQL misuses could lead into major security issues. Such execution on any remote connection, if the SQL statement is not a SELECT, is unsafe. In fact, if it may affect the data content.
By default, for security reasons, the {\f1\fs20 AllowRemoteExecute} field value in {\f1\fs20 SUPERVISOR_ACCESS_RIGHTS} constant does not include the {\f1\fs20 reSQL} flag. It means that no remote call will be allowed but for safe read-only SELECT statements.
When SELECT statements are sent, the server will always check for the table name specified in their FROM clause. If there is a single table appearing, its security policy will be checked against the {\f1\fs20 GET[]} flags of the corresponding table. If the SELECT statement is more complex (e.g. is a JOINed statement), then the {\f1\fs20 reSQLSelectWithoutTable} will be checked to ensure that the user has the right to execute such statements.
Another possibility of SQL remote execution is to add a {\f1\fs20 sql=....} inline parameter to a GET request (with optional paging). The {\f1\fs20 reUrlEncodedSQL} flag is used to enable or disable this feature.
Last but not least, a {\f1\fs20 WhereClause=...} inline parameter can be added to a DELETE request. The {\f1\fs20 reUrlEncodedDelete} option is used to enable or disable this feature.
You can change the default safe policy by including or excluding {\f1\fs20 reSQL}, {\f1\fs20 reSQLSelectWithoutTable}, {\f1\fs20 reUrlEncodedSQL} or {\f1\fs20 reUrlEncodedDelete} flags in the {\f1\fs20 TSQLAuthGroup.} {\f1\fs20 AccessRights}. {\f1\fs20 AllowRemoteExecute} field of an authentication user session.
If security is a real concern, you should enable @98@ and URI signature on your server, so that only trusted clients may access to the server. This is the main security rule of the framework - in practice, those {\i per table} access right or {\i SQL remote execution flags} are more a design rule than a strong security feature. Since remote execution of any SQL statements can be unsafe, we recommend to write a dedicated server-side service (method-based or interface-based) to execute such statements instead, and disallow remote SQL execution; then clients can safely use those dedicated safe services, and/or @*ORM@ @*CRUD@ operations for simple data requests. It will also help your project to be not tied to SQL, so that a switch to a {\i @*NoSQL@} persistence engine will still be possible, without changing the client code.
:   Service execution
The {\f1\fs20 reService} flag of {\f1\fs20 AllowRemoteExecute: TSQLAllowRemoteExecute} can be used to enable or unable the @63@ feature of {\i mORMot}.
In addition to this global parameter, you can set per-service and per-method @77@.
For @49@, if authentication is enabled, any method execution will be processed only from a signed URI.\line You can use {\f1\fs20 TSQLRestServer.ServiceMethodByPassAuthentication()} to disable the need of a signature for a given service method - e.g. it is the case for {\f1\fs20 Auth} and {\f1\fs20 Timestamp} standard method services.
For @63@, if authentication is enabled, any service execution will be processed only from a signed URI.\line You can use the {\f1\fs20 TServiceFactoryServer.ByPassAuthentication} property, to let a given service URI not be signed.
Do not forget to remove authentication for the services for which you want to enable @97@. In fact, such world-wide @*CDN@ caching services expect the URI to be generic, and not tied to a particular client session.
:79Scripting Engine
%cartoon04.png
: Scripting abilities
As a {\i Delphi} framework, {\i mORMot} premium language support is for the {\i object pascal} language. But it could be convenient to have some part of your software not fixed within the executable. In fact, once the application is compiled, execution flow is written in stone: you can't change it, unless you modify the {\i Delphi} source and compile it again. Since {\i mORMot} is {\i Open Source}, you can ship the whole source code to your customers or services with no restriction, and diffuse your own code as pre-compiled {\f1\fs20 .dcu} files, but your end-user will need to have a {\i Delphi} IDE installed (and paid), and know the {\i Delphi} language.
This is when @**script@ing does come on the scene.\line For instance, scripting may allow to customize an application behavior for an end-user (i.e. for reporting), or let a domain expert define evolving appropriate business rules - following @54@.
If your business model is to publish a core domain expertise (e.g. accounting, peripheral driving, database model, domain objects, communication, AJAX clients...) among several clients, you will sooner or later need to adapt your application to one or several of your customers. There is no "one {\f1\fs20 exe} to rule them all". Maintaining several executables could become a "branch-hell". Scripting is welcome here: speed and memory critical functionality (in which {\i mORMot} excels) will be hard-coded within the main executable, then everything else could be defined in script.
There are plenty of script languages available.\line We considered @http://code.google.com/p/dwscript which is well maintained and expressive (it is the code of our beloved {\i @*Smart Mobile Studio@}), but is not very commonly used. We still want to include it in the close future.\line Then @http://www.lua.org defines a light and versatile general-purpose language, dedicated to be embedded in any application. Sounds like a viable solution: if you can help with it, your contribution is welcome!\line We did also take into consideration @http://www.python.org and @http://www.ruby-lang.org but both are now far from light, and are not meant to be embedded, since they are general-purpose languages, with a huge set of full-featured packages.
Then, there is {\i JavaScript}:
- This is the {\i World Wide Web} assembler. Every programmer in one way or another knows {\i JavaScript};
- {\i JavaScript} can be a very powerful language - see Crockford's book "{\i JavaScript - The Good Parts}";
- There are a huge number of libraries written in {\i JavaScript}: template engines (jade, mustache...), SOAP and LDAP clients, and many others (including all {\f1\fs20 node.js} libraries of course);
- It was the base for some strongly-typed syntax extensions, like {\i CoffeScript, TypeScript, Dart};
- In case of {\i @*AJAX@} / {\i Rich Internet Application} we can directly share part of logic between client and server (validation, template rendering...) without any middle-ware;
- One long time {\i mORMot}'s user (Pavel, aka {\i mpv}) already integrated {\i SpiderMonkey} to {\i mORMot}'s core. His solution is used on production to serve billion of requests per day, with success. We officially integrated his units. Thanks Pavel!
As a consequence, {\i mORMot} introduced direct {\i JavaScript} support via {\i SpiderMonkey}.\line It allows to:
- Execute {\i Delphi} code from {\i JavaScript} - including our @13@/ @17@ methods, or even reporting;
- Consume {\i JavaScript} code from {\i Delphi} (e.g. to define and customize any service or rule, or use some existing {\f1\fs20 .js} library);
- Expose {\i JavaScript} objects and functions via a {\f1\fs20 @*TSMVariant@} custom variant type: it allows to access any {\i JavaScript} object properties or call any of its functions via @*late-binding@, from your {\i Delphi} code, just as if it was written in native Object-Pascal;
- Follow a classic synchronous blocking pattern, rooted on {\i mORMot}'s multi-thread efficient model, easy to write and maintain;
- Handle {\i JavaScript} or {\i Delphi} objects as @32@ @2@, ready to be published or consumed via @6@ remote access.
: SpiderMonkey integration
:  A powerful JavaScript engine
{\i @**SpiderMonkey@}, the Mozilla {\i @*JavaScript@} engine, can be embedded in your {\i mORMot} application. It could be used on client side, within a {\i Delphi} application (e.g. for reporting), but the main interest of it may be on the server side.
The word {\i JavaScript} may bring to mind features such as event handlers (like {\f1\fs20 onclick}), {\f1\fs20 DOM} objects, {\f1\fs20 window.open}, and {\f1\fs20 XMLHttpRequest}.\line But all of these features are actually not provided by the {\i SpiderMonkey} engine itself.
{\i SpiderMonkey} provides a few core {\i JavaScript} data types—numbers, strings, Arrays, Objects, and so on—and a few methods, such as {\f1\fs20 Array.push}. It also makes it easy for each application to expose some of its own objects and functions to {\i JavaScript} code. Browsers expose DOM objects. Your application will expose objects that are relevant for the kind of scripts you want to write. It is up to the application developer to decide what objects and methods are exposed to scripts.
:  Direct access to the SpiderMonkey API
The {\f1\fs20 SynSMAPI.pas} unit is a tuned conversion of the {\i SpiderMonkey} API, providing full {\i ECMAScript 5} support and {\i JIT}. The {\i SpiderMonkey} revision 24 engine is included, with a custom C wrapper around the original C++ code. \line You could take a look at @http://developer.mozilla.org/en-US/docs/Mozilla/Projects/SpiderMonkey for a full description of this low-level API, and find our patched version of the library, modified to be published from C instead of C++, in the {\f1\fs20 synsm-mozjs} folder of the {\i mORMot} source code repository.
But the {\f1\fs20 SynSM.pas} unit will encapsulate most of it into higher level {\i Delphi} classes and structures (including a custom {\f1\fs20 variant} type), so you probably won't need to use {\f1\fs20 SynSMAPI.pas} directly in your code:
|%25%75
|\b Type|Description\b0
|{\f1\fs20 TSMEngineManager}|main access point to the {\i SpiderMonkey} per-thread scripting engines
|{\f1\fs20 TSMEngine}|implements a Thread-Safe {\i JavaScript} engine instance
|{\f1\fs20 TSMObject}|wrap a {\i JavaScript} object and its execution context
|{\f1\fs20 TSMValue}|wrap a {\i JavaScript} value, and interfaces it with {\i Delphi} types
|{\f1\fs20 TSMVariant} /\line {\f1\fs20 TSMVariantData}|define a custom {\f1\fs20 variant} type, for direct access to any {\i JavaScript} object, with late-binding
|%
We will see know how to work with all those classes.
:  Execution scheme
The {\i SpiderMonkey JavaScript engine} compiles and executes scripts containing {\i JavaScript} statements and functions. The engine handles memory allocation for the objects needed to execute scripts, and it cleans up—garbage collects—objects it no longer needs.
In order to run any {\i JavaScript} code in {\i SpiderMonkey}, an application must have three key elements:
- A {\f1\fs20 JSRuntime},
- A {\f1\fs20 JSContext},
- And a {\i global} {\f1\fs20 JSObject}.
A {\f1\fs20 JSRuntime}, or runtime, is the space in which the JavaScript variables, objects, scripts, and contexts used by your application are allocated. Every {\f1\fs20 JSContext} and every object in an application lives within a {\f1\fs20 JSRuntime}. They cannot travel to other runtimes or be shared across runtimes.
A {\f1\fs20 JSContext}, or context, is like a little machine that can do many things involving {\i JavaScript} code and objects. It can compile and execute scripts, get and set object properties, call {\i JavaScript} functions, convert {\i JavaScript} data from one type to another, create objects, and so on.
Lastly, the {\i global} {\f1\fs20 JSObject} is a {\i JavaScript} object which contains all the classes, functions, and variables that are available for {\i JavaScript} code to use. Whenever a web browser code does something like {\f1\fs20 window.open("http://www.mozilla.org/")}, it is accessing a global property, in this case {\f1\fs20 window}. {\i SpiderMonkey} applications have full control over what global properties scripts can see.
Every {\i SpiderMonkey} instance starts out every execution context by creating its {\f1\fs20 JSRunTime}, {\f1\fs20 JSContext} instances, and a global {\f1\fs20 JSObject}. It populates this global object with the standard {\i JavaScript} classes, like {\f1\fs20 Array} and {\f1\fs20 Object}. Then application initialization code will add whatever custom classes, functions, and variables (like {\f1\fs20 window}) the application wants to provide; it may be, for a {\i mORMot} server application, ORM access or SOA services consumption and/or implementation.
Each time the application runs a {\i JavaScript} script (using, for example, {\f1\fs20 JS_EvaluateScript}), it provides the global object for that script to use. As the script runs, it can create global functions and variables of its own. All of these functions, classes, and variables are stored as properties of the global object.
:  Creating your execution context
The main point about those three key elements is that, in the current implementation pattern of {\i SpiderMonkey}, runtime, context or global objects are {\i not thread-safe}.
Therefore, in the {\i mORMot}'s use of this library, each thread will have its own instance of each.
In the {\f1\fs20 SynSM.pas} unit, a {\f1\fs20 TSMEngine} class has been defined to give access to all those linked elements:
!  TSMEngine = class
!  ...
!    /// access to the associated global object as a TSMVariant custom variant
!    // - allows direct property and method executions in Delphi code, via
!    // late-binding
!    property Global: variant read FGlobal;
!    /// access to the associated global object as a TSMObject wrapper
!    // - you can use it to register a method
!    property GlobalObject: TSMObject read FGlobalObject;
!    /// access to the associated global object as low-level PJSObject
!    property GlobalObj: PJSObject read FGlobalObject.fobj;
!    /// access to the associated execution context
!    property cx: PJSContext read fCx;
!    /// access to the associated execution runtime
!    property rt: PJSRuntime read frt;
!  ...
Our implementation will define one Runtime, one Context, and one global object per thread, i.e. one {\f1\fs20 TSMEngine} class instance per thread.
A {\f1\fs20 JSRuntime}, or runtime, is created for each {\f1\fs20 TSMEngine} instance. In practice, you won't need access to this value, but rely either on a {\f1\fs20 JSContext} or directly a {\f1\fs20 TSMEngine}.
A {\f1\fs20 JSContext}, or context, will be the main entry point of all {\i SpiderMonkey} API, which expect this context to be supplied as parameter. In mORMot, you can retrieve the running {\f1\fs20 TSMEngine} from its context by using the {\f1\fs20 function TSMObject.Engine: TSMEngine} - in fact, the engine instance is stored in the {\i private data} slot of each {\f1\fs20 JSContext}.
Lastly, the {\f1\fs20 TSMEngine}'s {\i global object} contains all the classes, functions, and variables that are available for JavaScript code to use. For a {\i mORMot} server application, ORM access or SOA services consumption and/or implementation, as stated above.
You can note that there are several ways to access this global object instance, from high-level to low-level {\i JavaScript} object types. The {\f1\fs20 TSMEngine.Global} property above is in fact a {\f1\fs20 variant}. Our {\f1\fs20 SynSM.pas} unit defines in fact a custom {\f1\fs20 variant} type, identified as the {\f1\fs20 @**TSMVariant@ class}, able to access any JavaScript object via @*late-binding@, for both variables and functions:
!  engine.Global.MyVariable := 1.0594631;
!  engine.Global.MyFunction(1,'text');
Most web applications only need one runtime, since they are running in a single thread - and (ab)use of callbacks for non-blocking execution. But in {\i mORMot}, you will have one {\f1\fs20 TMSEngine} instance per thread, using the {\f1\fs20 TSMEngineManager.ThreadSafeEngine} method. Then all execution may be blocking, without any noticeable performance issue, since the whole {\i mORMot} threading design was defined to maximize execution resources.
:  Blocking threading model
This threading model is the big difference with other server-side scripting implementation schemes, e.g. the well-known {\f1\fs20 @*node.js@} solution.
Multi-threading is not evil, when properly used. And thanks to the {\i mORMot}'s design, you won't be afraid of writing {\i blocking} JavaScript code, without any callbacks. In practice, those callbacks are what makes most {\i JavaScript} code difficult to maintain.
On the client side, i.e. in a web browser, the {\i JavaScript} engine only uses one thread per web page, then uses callbacks to defer execution of long-running methods (like a remote HTTP request).\line If fact, this is one well identified performance issue of modern AJAX applications. For instance, it is not possible to perform some intensive calculation in {\i JavaScript}, without breaking the web application responsiveness: you have to split your computation task in small tasks, then let the {\i JavaScript} code pause, until a next piece of computation could be triggerred... On server side, {\f1\fs20 node.js} allows to define {\i Fibers} and {\i Futures} - see @http://github.com/laverdet/node-fibers - but this is not available on web clients. Some browsers did only start to uncouple the {\i JavaScript} execution thread from the HTML rendering thread - and even this is hard to implement... we reached here the limit of a technology rooted in the 80's...
On the server side, {\f1\fs20 node.js} did follow this pattern, which did make sense (it allows to share code with the client side, with some name-space tricks), but it is also a big waste of resources. Why should we stick to an implementation pattern inherited from the 80's computing model, when all CPUs were mono core, and threads were not available?
The main problem when working with one single thread, is that your code shall be asynchronous. Soon or later, you will face a syndrome known as "{\i Callback Hell}". In short, you are nesting anonymous functions, and define callbacks. The main issue, in addition to lower readability and being potentially sunk into {\f1\fs20 function()} nesting, is that you just lost the {\i JavaScript} exception model. In fact, each callback function has to explicitly check for the error (returned as a parameter in the callback function), and handle it.
Of course, you can use so-called {\i Promises} and some nice libraries - mainly {\f1\fs20 async.js}.\line But even those libraries add complexity, and make code more difficult to write. For instance, consider the following non-blocking/asynchronous code:
#getTweetsFor("domenic") // promise-returning function
#  .then(function (tweets) {
#    var shortUrls = parseTweetsForUrls(tweets);
#    var mostRecentShortUrl = shortUrls[0];
#    return expandUrlUsingTwitterApi(mostRecentShortUrl); // promise-returning function
#  })
#  .then(httpGet) // promise-returning function
#  .then(
#    function (responseBody) {
#      console.log("Most recent link text:", responseBody);
#    },
#    function (error) {
#      console.error("Error with the twitterverse:", error);
#    }
#  );
Taken from @https://blog.domenic.me/youre-missing-the-point-of-promises/
This kind of code will be perfectly readable for a {\i JavaScript} daily user, or someone fluent with functional languages.
But the following blocking/synchronous code may sound much more familiar, safer and less verbose, to most {\i Delphi} / Java / C# programmer:
#try {
#  var tweets = getTweetsFor("domenic"); // blocking
#  var shortUrls = parseTweetsForUrls(tweets);
#  var mostRecentShortUrl = shortUrls[0];
#  var responseBody = httpGet(expandUrlUsingTwitterApi(mostRecentShortUrl)); // blocking x 2
#  console.log("Most recent link text:", responseBody);
#} catch (error) {
#  console.error("Error with the twitterverse: ", error);
#}
Thanks to the blocking pattern, it becomes obvious that code readability and maintainability is as high as possible, and error detection is handled nicely via {\i JavaScript} exceptions, and a global {\f1\fs20 try .. catch}.
Last but not least, debugging blocking code is easy and straightforward, since the execution will be linear, following the code flow.
Upcoming ECMAScript 6 should go even further thanks to the {\f1\fs20 yield} keyword and some task generators - see @http://taskjs.org - so that asynchronous code may become closer to the synchronous pattern. But even with {\f1\fs20 yield}, your code won't be as clean as with plain blocking style.
In {\i mORMot}, we did choose to follow an alternate path, i.e. write blocking synchronous code. Sample above shows how easier it is to work with. If you use it to define some huge business logic, or let a domain expert write the code, blocking syntax is much more straightforward.
Of course, {\i mORMot} allows you to use callbacks and functional programming pattern in your {\i JavaScript} code, if needed. But by default, you are allowed to write KISS blocking code.
: Interaction with existing code
Within {\i mORMot} units, you can mix {\i Delphi} and {\i JavaScript} code by two ways:
- Either define your own functions in {\i Delphi} code, and execute them from {\i JavaScript};
- Or define your own functions in {\i JavaScript} code (including any third-party library), and execute them from {\i Delphi}.
Like for other part of our framework, performance and integration has been tuned, to follow our @*KISS@ way.
You can take a look at "{\f1\fs20 22 - JavaScript HTTPApi web server\\JSHttpApiServer.dpr}" sample for reference code.
:  Proper engine initialization
As was previously stated, the main point to interface the {\i JavaScript} engine is to {\i register} all methods when the {\f1\fs20 @*TSMEngine@} instance is initialized.
For this, you set the corresponding {\f1\fs20 OnNewEngine} callback event to the main {\f1\fs20 @*TSMEngineManager@} instance.\line See for instance, in the sample code:
!constructor TTestServer.Create(const Path: TFileName);
!begin
!  ...
!  fSMManager := TSMEngineManager.Create;
!  fSMManager.OnNewEngine := DoOnNewEngine;
!  ...
In {\f1\fs20 DoOnNewEngine}, you will initialize every newly created {\f1\fs20 TSMEngine} instance, to register all needed {\i Delphi} methods and prepare access to {\i JavaScript} via the runtime's global {\f1\fs20 JSObject}.
Then each time you want to access the {\i JavaScript} engine, you will write for instance:
!function TTestServer.Process(Ctxt: THttpServerRequest): cardinal;
!var engine: TSMEngine;
!...
!   engine := fSMManager.ThreadSafeEngine;
!...  // now you can use engine, e.g. engine.Global.someMethod()
Each thread of the HTTP server thread-pool will be initialized on the fly if needed, or the previously initialized instance will be quickly returned otherwise.
Once you have the {\f1\fs20 TSMEngine} instance corresponding to the current thread, you can launch actions on its global object, or tune its execution.\line For instance, it could be a good idea to check for the {\i JavaScript} VM's @*garbage collector@:
!function TTestServer.Process(Ctxt: THttpServerRequest): cardinal;
!...
!   engine := fSMManager.ThreadSafeEngine;
!   engine.MaybeGarbageCollect; // perform garbage collection if needed
!...
We will now find out how to interact between {\i JavaScript} and {\i Delphi} code.
:  Calling Delphi code from JavaScript
In order to call some {\i Delphi} method from {\i JavaScript}, you will have to register the method.\line As just stated, it is done by setting a callback within {\f1\fs20 TSMEngineManager.OnNewEngine} initialization code. For instance:
!procedure TTestServer.DoOnNewEngine(const Engine: TSMEngine);
!...
!  // add native function to the engine
!  Engine.RegisterMethod(Engine.GlobalObj,'loadFile',LoadFile,1);
!end;
Here, the local {\f1\fs20 LoadFile()} method is implemented as such in native code:
!function TTestServer.LoadFile(const This: variant; const Args: array of variant): variant;
!begin
!  if length(Args)<>1 then
!    raise Exception.Create('Invalid number of args for loadFile(): required 1 (file path)');
!  result := AnyTextFileToSynUnicode(Args[0]);
!end;
As you can see, this is perfectly easy to follow.\line Its purpose is to load a file content from {\i JavaScript}, by defining a new global function named {\f1\fs20 loadFile()}.\line Remember that the {\i SpiderMonkey} engine, by itself, does not know anything about file system, database or even DOM. Only basic objects were registered, like arrays. We have to explicitly register the functions needed by the {\i JavaScript} code.
In the above code snippet, we used the {\f1\fs20 TSMEngineMethodEventVariant} callback signature, marshalling {\f1\fs20 variant} values as parameters. This is the easiest method, with only a slight performance impact.
Such methods have the following features:
- Arguments will be transmitted from {\i JavaScript} values as simple {\i Delphi} types (for numbers or text), or as our custom {\f1\fs20 TSMVariant} type for {\i JavaScript} objects, which allows @*late-binding@;
- The {\f1\fs20 This: variant} first parameter map the "callee" {\i JavaScript} object as a {\f1\fs20 TSMVariant} custom instance, so that you will be able to access the other object's methods or properties directly via late-binding;
- You can benefit of the {\i JavaScript} feature of variable number of arguments when calling a function, since the input arguments is a dynamic array of {\f1\fs20 variant};
- All those registered methods are registered in a list maintained in the {\f1\fs20 TSMEngine} instance, so it could be pretty convenient to work with, in some cases;
- You can still access to the low-level {\i JSObject} values of any the argument, if needed, since they can be trans-typed to a {\f1\fs20 TSMVariantData} instance (see below) - so you do not loose any information;
- The {\i Delphi} native method will be protected by the {\i mORMot} wrapper, so that any exception raised within the process will be catch and transmitted as a {\i JavaScript} exception to the runtime;
- There is also an hidden set of the FPU exception mask during execution of native code (more on it later on) - you should not bother on it here.
Now consider how you should have written the same {\f1\fs20 loadFile()} function via low-level API calls.
First, we register the callback:
!procedure TTestServer.DoOnNewEngine(const Engine: TSMEngine);
!...
!  // add native function to the engine
! Engine.GlobalObject.DefineNativeMethod('loadFile', nsm_loadFile, 1);
!end;
Then its implementation:
!function nsm_loadFile(cx: PJSContext; argc: uintN; vp: Pjsval): JSBool; cdecl;
!var in_argv: PjsvalVector;
!    filePath: TFileName;
!begin
!  TSynFPUException.ForDelphiCode;
!  try
!    if argc<>1 then
!      raise Exception.Create('Invalid number of args for loadFile(): required 1 (file path)');
!    in_argv := JS_ARGV(cx,vp);
!    filePath := JSVAL_TO_STRING(in_argv[0]).ToString(cx);
!    JS_SET_RVAL(cx, vp, cx^.NewJSString(AnyTextFileToSynUnicode(filePath)).ToJSVal);
!    Result := JS_TRUE;
!  except
!    on E: Exception do begin // all exceptions MUST be catched on Delphi side
!      JS_SET_RVAL(cx, vp, JSVAL_VOID);
!      JSError(cx, E);
!      Result := JS_FALSE;
!    end;
!  end;
!end;
As you can see, this {\f1\fs20 nsm_loadFile()} function is much more difficult to follow:
- Your code shall begin with a cryptic {\f1\fs20 TSynFPUException.ForDelphiCode} instruction, to protect the FPU exception flag during execution of native code ({\i Delphi} RTL expects its own set of FPU exception mask during execution, which does not match the FPU exception mask expected by {\i SpiderMonkey});
- You have to explicitly catch any {\i Delphi} exception which may raise, with a {\f1\fs20 try...finally} block, and marshal them back as {\i JavaScript} errors;
- You need to do a lot of manual low-level conversions - via {\f1\fs20 JS_ARGV()} then e.g. {\f1\fs20 JSVAL_TO_STRING()} macros - to retrieve the actual values of the arguments;
- And the returning function is to be marshaled by hand - see the {\f1\fs20 JS_SET_RVAL()} line.
Since the {\f1\fs20 variant}-based callback has only a slight performance impact (nothing measurable, when compared to the {\i SpiderMonkey} engine performance itself), and still have access to all the transmitted information, we strongly encourage you to use this safer and cleaner pattern, and do not define any native function via low-level API.
Note that there is an alternate JSON-based callback, which is not to be used in your end-user code, but will be used when marshalling to JSON is needed, e.g. when working with {\i mORMot}'s ORM or SOA features.
:  TSMVariant custom type
As stated above, the {\f1\fs20 SynSM.pas} unit defines a {\f1\fs20 @**TSMVariant@} custom variant type. It will be used by the unit to marshal any {\i JSObject} instance as variant.
Via the magic of @*late-binding@, it will allow access of any {\i JavaScript} object property, or execute any of its functions. Only with a slightly performance penalty, but with much better code readability than with low-level access of the {\i SpiderMonkey} API.
The {\f1\fs20 TSMVariantData} memory structure can be used to map such a {\f1\fs20 TSMVariant} variant instance. In fact, the custom variant type will store not only the {\i JSObject} value, but also its execution context - i.e. {\i JSContext} - so is pretty convenient to work with.
For instance, you may be able to write code as such:
!function TMyClass.MyFunction(const This: variant; const Args: array of variant): variant;
!var global: variant;
!begin
!  TSMVariantData(This).GetGlobal(global);
!  global.anotherFunction(Args[0],Args[1],'test');
!  // same as:
!  global := TSMVariantData(This).SMObject.Engine.Global;
!  global.anotherFunction(Args[0],Args[1],'test');
!  // but you may also write directly:
!  with TSMVariantData(This).SMObject.Engine do
!    Global.anotherFunction(Args[0],Args[1],'test');
!  result := AnyTextFileToSynUnicode(Args[0]);
!end;
Here, the {\f1\fs20 This} custom variant instance is trans-typed via {\f1\fs20 TSMVariantData(This)} to access its internal properties.
:  Calling JavaScript code from Delphi
In order to execute some {\i JavaScript} code from {\i Delphi}, you should first define the {\i JavaScript} functions to be executed.\line This shall take place within {\f1\fs20 TSMEngineManager.OnNewEngine} initialization code:
!procedure TTestServer.DoOnNewEngine(const Engine: TSMEngine);
!var showDownRunner: SynUnicode;
!begin
!  // add external JavaScript library to engine (port of the Markdown library)
!  Engine.Evaluate(fShowDownLib, 'showdown.js');
!  // add the bootstrap function calling loadfile() then showdown's makeHtml()
!  showDownRunner := AnyTextFileToSynUnicode(ExeVersion.ProgramFilePath+'showDownRunner.js');
!  Engine.Evaluate(showDownRunner, 'showDownRunner.js');
!  ...
This code first {\i evaluates} (i.e. "executes") a general-purpose {\i JavaScript} library contained in the {\f1\fs20 showdown.js} file, available in the sample executable folder. This is an open source library able to convert any {\i Markdown} markup into HTML. Plain standard {\i JavaScript} code.
Then we {\i evaluate} (i.e. "execute") a small piece of {\i JavaScript} code, to link the {\f1\fs20 makeHtml()} function of the just defined library with our {\f1\fs20 loadFile()} native function:
#function showDownRunner(pathToFile){
#  var src = loadFile(pathToFile);            // call Delphi native code
#  var converter = new Showdown.converter();  // get the Showdown converted
#  return converter.makeHtml(src);            // convert .md content into HTML via showdown.js
#}
Now we have a new global {\f1\fs20 function showDownRunner(pathToFile)} at hand, ready to be executed by our {\i Delphi} code:
!function TTestServer.Process(Ctxt: THttpServerRequest): cardinal;
!var content: variant;
!    FileName, FileExt: TFileName;
!    engine: TSMEngine;
!  ...
!  if FileExt='.md' then begin
!  ...
!    engine := fSMManager.ThreadSafeEngine;
!  ...
!    content := engine.Global.showDownRunner(FileName);
!  ...
As you can see, we access the function via @*late-binding@. Above code is perfectly readable, and we call here a {\i JavaScript} function and a whole library as natural as if it was native code.
Without late-binding, we may have written, accessing not the {\f1\fs20 Global TSMVariant} instance, but the lower level {\f1\fs20 GlobalObject: TSMObject} property:
!  ...
!    content := engine.GlobalObject.Run('showDownRunner',[SynUnicode(FileName)]);
!  ...
It is up to you to choose which kind of code you prefer, but late-binding is worth considering.
:187Asymmetric Encryption
%cartoon05.png
As we have seen when dealing about @43@, the framework offers built-in @**encryption@ of the content transmitted between its @*REST@ client and server sides, especially via @188@, or @*HTTPS@. The later, when using TLS 1.2 and proven patterns, implements state-of-the-art security. But default {\i mORMot} encryption, even if using proven algorithms like @*AES@256-CFB and @*SHA256@, uses symmetric keys, that is the same secret key is shared on both client and server sides.
@**Asymmetric@ encryption, also known as @**public-key@ cryptography, uses pairs of keys:
- {\i Public} keys that may be disseminated widely;
- Paired with {\i private} keys which are known only to the owner.
The framework features a full asymmetric encryption system, based on {\i @*Elliptic curve@ cryptography} (@**ECC@), which may be used at application level (i.e. to protect your application data), or at transmission level (to enhance communication safety).
\page
: Public-key Cryptography
Once you have {\i generated} a public/private pairs of keys, you can perform two functions:
- {\i Authenticate} a message originated with a holder of the private key; a {\i certification} system should be used to maintain a trust chain of authority;
- {\i Encrypt} a message with a public key to ensure that only the holder of the paired private key can decrypt it.
:  Keys Generation and Distribution
First process is to generate a pair of public/private keys. Some random number generator, probably based on an external entropy source, will gather unpredictable numbers, which will be consumed by a public-key algorithm to generate the actual set of keys. This step usually requires some computing powers, due to the complexity of the algorithms involved, and the encryption needed for storing the private key in secret.
Let's explain how it works for the classic Alice/Bob scheme:
\graph AsymmKeygen Asymmetric Key Generation
\Random Seed\Key Generation Algorithm
\Key Generation Algorithm\alice.public
\Key Generation Algorithm\alice.private
\Random Seed \Key Generation  Algorithm
\Key Generation  Algorithm\bob.public
\Key Generation  Algorithm\bob.private
\
Now we have two pairs of keys:
- {\f1\fs20 alice.public} and {\f1\fs20 alice.private} for Alice;
- {\f1\fs20 bob.public} and {\f1\fs20 bob.private} for Bob.
By design, {\i public} keys ({\f1\fs20 alice.public} and {\f1\fs20 bob.public}) can be published, via mail, in application settings, as unprotected file, or even on a public server. On the contrary, {\i private} keys ({\f1\fs20 alice.private} or {\f1\fs20 bob.private}) should remain as secret as possible, and are usually encrypted, then stored in password-protected files, in some safe place of the operating system, or even dedicated hardware.
In practice, Alice will send her {\f1\fs20 alice.public} key to Bob, so that:
- Bob can verify the digital signature of a message sent by Alice, who signed it with her {\f1\fs20 alice.private} key;
- Bob can encrypt some information with the known {\f1\fs20 alice.public} key, then send it to Alice - and that only Alice could decrypt it using her {\f1\fs20 alice.private} key.
Of course, since Bob has his own set of keys, he also publishes his {\f1\fs20 bob.public} key, so that:
- Alice can verify the digital signature of a message sent by Blob, who signed it with his {\f1\fs20 bob.private} key;
- Alice can encrypt some information with the known {\f1\fs20 bob.public} key, then send it to Bob - and that only Bob could decrypt it using his {\f1\fs20 bob.private} key.
Key distribution is an important part of any asymmetric encryption scheme. The whole security chain is as secure as its weakest link, so the secrecy of the private keys requires as most attention as possible. Every software solution using security will probably require external audits, at least peer review, to validate each implementation.
:  Message Authentication
Any kind of message (most probably a file or a memory buffer) can be authenticated using {\i @*digital signature@s}, using the private key of the sender. Then, on the other side, the receiver can verify the message signature, using the public key of the sender.
\graph AsymmSign Asymmetric Digital Signature
subgraph cluster_0 {
label="Alice";
\alice.private\Signing Algorithm
\message.txt\Signing Algorithm
\Signing Algorithm\message.sign
}
subgraph cluster_1 {
label="Bob";
\message.txt \Verifying Algorithm
\message.sign \Verifying Algorithm
\alice.public\Verifying Algorithm
\Verifying Algorithm\accepted\  ?
\Verifying Algorithm\rejected
}
\
As you can see, if Bob believes that the {\f1\fs20 alice.public} file comes from Alice, he can assume that the {\f1\fs20 messate.txt} content has really been sent by Alice. Most of the time, in such simple scenarios, Alice probably gave directly her {\f1\fs20 alice.public} file to Bob, for instance via an email. But for most complex scenarios, like the Client/Server solutions which can be built using the {\i mORMot} framework, the multiplicity of nodes, and therefore keys, induces a potential risk.
:  Certificates and Public Key Infrastructure
A central problem with the use of public-key cryptography is confidence that a particular public key is authentic, in that it belongs to the person or entity claimed, and has not been tampered with or replaced by a malicious third party. Digital signature is more than just creating a hash of some content, or applying some kind of "seal" on it: validation should be done against some reference public keys, which are hosted into a {\i public-key infrastructure} ({\f1\fs20 @*PKI@}). One or more third-parties, aka certification authorities ({\f1\fs20 @*CA@}), certify ownership of key pairs, by supplying some online service and/or local safe storage of reference public keys, but keeping their own private keys secret. Any certificate authority could sign a message with his private key, or even delegate his own authority to another certification instance, by signing the intermediate authority with his private key. If one certificate is compromised - i.e. if its private key has been released - the whole chain of trust is broken, and all dependent certificates should be immediately revoked.
In practice, when a public certificate key is generated in such a trusted {\f1\fs20 PKI} system, it will contain:
- The genuine public key material, depending on the underlying algorithm used;
- Some ownership information (i.e. who emitted it);
- The scope of the certification (may apply to a user, a company, a web site, an application...);
- A certified link to one or several other certificates, signed with their private key to prove their authenticity using the known public key of the {\f1\fs20 CA} chain;
- Optional validity and revocation dates - since it is a good practice to renew certificates on a regular basis.
The private key store may also contain the very same set of information, added to its private key material. It will enforce consistency between public and private keys - for instance, you won't be fooled by using a private key after its associated public certificates expired.
Certification authorities create a chain of trust, used as reference to authenticate public keys. Every Operating System, or Internet browser do contain some root certificates, and the whole Internet security (HTTPS/TLS) is governed by such a {\f1\fs20 PKI}. Of course, for your own set of applications or products, you can create your own key chain, keeping the same principles - mainly private key secrecy and trust chain management.
:  Message encryption
A naive approach of hiding a message content is by using a secret key or pattern, then apply it on the message. It has been done since ages, and will be safe as much as the symmetric key is safe. As a side effect, you have to trust the receiver not to spread the key to the public - and in fact, you shouldn't: don't trust anyone, even not you!
Public-key cryptography solves this problem by using a public key to encrypt a message, which will therefore only be decryptable by someone knowing the corresponding secret key.
\graph AsymmEncrypt Asymmetric Encryption Scheme
subgraph cluster_0 {
label="Bob";
\alice.public\Encryption Algorithm
\message.txt\Encryption Algorithm
\Encryption Algorithm\message.encrypted
}
subgraph cluster_1 {
label="Alice";
\message.encrypted \Decryption Algorithm
\alice.private\Decryption Algorithm
\Decryption Algorithm\ message.txt
}
\
Of course, you can not only encrypt the message, but also sign it, using the other end public key. Here is how a sign-then-encrypt pattern can be implemented:
\graph AsymmSignEncrypt Asymmetric Sign-Then-Encrypt Scheme
subgraph cluster_0 {
label="Bob";
\bob.private\Signing Algorithm
\message.txt\Signing Algorithm
\Signing Algorithm\message.sign
\alice.public\Encryption Algorithm
\message.txt\Encryption Algorithm
\message.sign\Encryption Algorithm
\Encryption Algorithm\message.signencrypted
}
subgraph cluster_1 {
label="Alice";
\message.signencrypted \Decryption Algorithm
\alice.private\Decryption Algorithm
\Decryption Algorithm\ message.txt
\message.signencrypted \Verifying Algorithm
\ message.txt\Verifying Algorithm
\bob.public\Verifying Algorithm
\Verifying Algorithm\accepted\  ?
\Verifying Algorithm\rejected
}
\
As always, the {\f1\fs20 alice.public} and {\f1\fs20 bob.public} keys are validated against the trust chain of certificates of a {\i public-key infrastructure} ({\f1\fs20 @*PKI@}).
With all this elements, we can now apply them to our {\i mORMot} applications.
\page
: Elliptic Curve Cryptography
The framework features an implementation of {\i Elliptic Curve Cryptography} (@**ECC@), based on the mathematical structure of the "elliptic curve discrete logarithm problem". The mathematical community has not made any major progress in improving algorithms to solve this problem since it was independently introduced by Koblitz and Miller in 1985. In short, the public key is an equation for an elliptic curve and a point that lies on that curve. The private key is a number. Thanks to the symmetry of the elliptic curve, there is some kind of symmetry also between ECC public and private key values, and ECDSA and ECDH algorithms capitalize on this characteristic to compute a digital signature or a shared secret.
In comparison to the RSA algorithm, ECC has some advantages:
- Smaller key size, for the same level of safety (a 256-bit elliptic curve key is comparable to a 3072-bit RSA key);
- Well endorsed by most certification authorities (NIST/NSA);
- Faster performance, especially when the key size increases;
- Offers @*perfect forward secrecy@, since a fresh key is created for every encryption;
- Potentially less patents infringement, in all its practical appliances;
- Last but not least, it is one the strongest algorithms for the future of web.
There will no doubt be criticism of our decision to re-implement a whole public-key cryptography stack from scratch, with its own small choice of algorithms, instead of using an existing library (like OpenSSL), and established standards (like X509).\line To be fair, such libraries are complex and confusing, whereas we selected a set of future-proof algorithms (AES256 excluding ECB, @*HMAC-SHA256@, @*PBKDF2_HMAC_SHA256@, ECDSA, ECIES...) to follow {\i mORMot}'s KISS and DRY principles, keep code maintainable and readable, and reduce risk assessment scope. We followed all identified best practices, and tried to avoid, from the beginning, buffer overflows, weak protocols, low entropy, low default values, serial collision, forensic vulnerabilities, hidden memory copies, evil optimizations. The last thing we want to do is to start mandating DLLs, which are perhaps deprecated/unsafe if part of the OS. Last but not least, it was fun, we learned a lot, and we hope you will enjoy using it, and contribute to it!
;- We did not start from scratch, since we used another proven Open Source library for the raw ECC computation, which was the most sensitive part;
;- Existing librairies have to deal with a lot of algorithms, options and old features: we wanted a reduced scope, to ease risk assessment - only well-known and future-proof algorithms were selected (AES256 excluding ECB, HMAC_SHA256, PBKDF2_HMAC_SHA256, ECDSA, ECIES...) and default values are very aggressive (password strength, 60,000 PBKDF2 iterations...);
;- Existing libraries are so complex that interfacing with them makes the consuming code complex to write and maintain - {\f1\fs20 SynEcc} logic is implemented in a few dozen lines of code: most of the unit source is about wrapper methods and documentation, and an average programmer can understand and review it, even if he/she is no Delphi expert;
;- A new implementation may benefit from past issues: we followed all identified best practices, and tried to avoid, from the beginning, buffer overflows, weak protocols, low entropy, low default values, serial collision, forensic vulnerabilities, hidden memory copies, evil optimizations;
;- It integrates nicely with other {\i mORMot} features, and re-uses the {\f1\fs20 SynCrypto.pas} unit for actual cryptography on all supported platforms, so the development effort was not big, and the resulting executables size did not increase;
;- As always, we started by writing tests, and we have pretty good automated tests coverage, from low-level ECC functions up to the highest level (we even validate the ECC command line tool);
;- We forbid file stamping, preferred JSON to any other text format, and used fixed sized binary buffers (e.g. for identifiers), with all-inclusive information, to avoid memory copies of sensitive data and logic flows depending on the feature set;
;- Some unique features were introduced (like AFSpliting or enforcing passwords for private keys), and in doubt, we always did choose the paranoid solution;
;- We are proud that {\i mORMot} applications are stand-alone executables, so the last thing we want to do is to start mandating DLLs, or be coupled to a specific Operating System;
;- Having our own embedded code is safer than using the old/unsafe already installed libraries, especially on an existing server (what is the OpenSSL version in your good old Debian VM?);
;- It was fun, we learned a lot, and we hope you will enjoy using it, and contribute to it!
:  Introducing SynEcc
The {\i mORMot}'s {\f1\fs20 SynEcc.pas} unit implements full ECC computation, using {\f1\fs20 secp256r1} curve, i.e. {\f1\fs20 NIST P-256}, or OpenSSL's {\f1\fs20 prime256v1}. The low-level computation is done in optimized C code - from the @https://github.com/esxgx/easy-ecc {\i Open Source} project - and is statically linked in your Windows or Linux executable: i.e. no external {\f1\fs20 .dll}/{\f1\fs20 .so} library is needed. On targets (e.g. BSD/MacOSX or ARM) where we didn't provide the static {\f1\fs20 .o} files, there is an optimized pascal version available. Then we defined a feature-rich set of object pascal classes on top of this solid ECC ground, to include certificates, safe storage of private keys, JSON publication of public keys, as an integrated toolset.
All needed low-level asymmetric cryptography is available:
- ECC key generation, using {\f1\fs20 SynCrypto.pas}'s secure {\f1\fs20 TAESPRNG} as random seed;
- ECDSA signature and verification of 256-bit hashes;
- ECDSH shared secret computation - suitable for ECIES encryption, after @*PBKDF2_HMAC_SHA256@ derivation.
The very same {\f1\fs20 SynEcc.pas} unit defines some high-level classes and structures, ready to implement:
- Authority certificates - via public {\f1\fs20 TECCCertificate} and private {\f1\fs20 TECCCertificateSecret} classes, and full {\f1\fs20 PKI} chaining - see {\f1\fs20 TECCCertificateChain};
- Digital signature of files or memory buffers - via {\f1\fs20 TECCSignatureCertified};
- Encryption of files or memory buffers - calling {\f1\fs20 TECCCertificate.Encrypt} and {\f1\fs20 TECCCertificateSecret.Decrypt} methods;
- Innovative {\f1\fs20 .cheat} files generation, for safe storage of private keys passwords, encrypted from a master {\f1\fs20 cheat.public} key and its master password.
You are free to use those classes, in your programs, whenever some advanced cryptography is needed - and it will eventually be the case, trust me! A command-line {\f1\fs20 ECC} tool has also been developed, for convenient operation on files.
:  ECC command line tool
You will find in the {\f1\fs20 SQLite3\\Samples\\33 - ECC} folder the source code of the {\f1\fs20 @*ECC@.dpr} console project. Just compile it into an executable, accessible from your command line prompt. Or download an already compiled version from @https://synopse.info/files/ecc.7z
It works with no problem under @*Windows@, or @*Linux@, with no external dependency (e.g. no {\i OpenSSL} needed), so could be used in an automated server infrastructure. No need to deploy a complex @*PKI@ system, just manage your certificates, encryption and signature details, via a single command line tool.
If you run it without argument, you will get simple help information (here is the list at the time of this writing, your own version may differ):
$>ecc
$
$Synopse ECC certificate-based public-key cryptography
$-----------------------------------------------------
$Using mORMot's SynECC rev. 1.18.3112
$
$ECC help
$ECC new -auth key.private -authpass authP@ssW0rd -authrounds 60000
$      -issuer toto@toto.com -start 2016-10-30 -days 30
$      -newpass P@ssw0RD@ -newrounds 60000
$ECC rekey -auth key.private -authpass P@ssW0rd -authrounds 60000
$      -newpass newP@ssw0RD@ -newrounds 60000
$ECC sign -file some.doc -auth key.private -pass P@ssW0rd -rounds 60000
$ECC verify -file some.doc -auth key.public
$ECC source -auth key.private -pass P@ssW0rd -rounds 60000
$      -const MY_PRIVKEY -comment "My Private Key"
$ECC infopriv -auth key.private -pass P@ssW0rd -rounds 60000
$ECC chain file1.public file2.public file3.public ...
$ECC chainall
$ECC crypt -file some.doc -out some.doc.synecc -auth key.public
$      -saltpass salt -saltrounds 60000
$ECC decrypt -file some.doc.synecc -out some.doc -auth key.private
$      -authpass P@ssW0rd -authrounds 60000 -saltpass salt -saltrounds 60000
$ECC infocrypt -file some.doc.synecc
$ECC cheatinit -newpass MasterP@ssw0RD@ -newrounds 100000
$ECC cheat -auth key.private -authpass MasterP@ssw0RD@ -authrounds 100000
$
$Note that you can add the -noprompt switch for no console interactivity.
$
As you can see, the action is defined by a keyword, at first place ({\f1\fs20 new sign verify source}...). Then some optional parameters, in form of {\f1\fs20 -key value} pairs, can be supplied. If no parameter is specified, the {\f1\fs20 ECC} console application will prompt for input, with user-friendly questions, and adequate default values.
You can define the {\f1\fs20 -noprompt} switch to force no console interaction at all, therefore allowing automated use from another process, or batch file. The {\f1\fs20 ECCProcess.pas} unit publishes all high-level commands of the {\f1\fs20 ECC} tool, so could be reused in your own setup or maintenance projects.
We will now use this {\f1\fs20 ECC} tool to show most common features of the {\f1\fs20 SynEcc} unit, but also showing the code corresponding to each action.
:  Keys and Certificates Generation
The first step is to create a new key pair, which will contain their own certification information:
$>ecc new
$Enter the first chars of the .private file name of the signing authority.
$Will create a self-signed certificate if left void.
$Auth:
$
$Enter Issuer identifier text.
$Will be truncated to 15-20 ascii-7 chars.
$Issuer [arbou] :
$
$Enter the YYYY-MM-DD start date of its validity.
$0 will create a never-expiring certificate
$Start [2016-09-23] :
$
$Enter the number of days of its validity.
$Days [365] :
$
$Enter a private PassPhrase for the new key (at least 8 chars long).
$Save this in a safe place: if you forget it, the key will be useless!
$NewPass [#weLHn5E.Qfe] :
$
$Enter the PassPhrase iteration round for the new key (at least 1000).
$The higher, the safer, but will demand more computation time.
$NewRounds [60000] :
$
$Corresponding TSynPersistentWithPassword.ComputePassword:
$ encryption ErHdwwro/8jFsCZC
$ authMutual 5qMgx6Miv+O71+VYL95zk6U2wP79lKL3s1BFnd+a
$ authServer 5qMhx6Miv+O71+VYL95zk6U2wP79lKL3s1BFnd+a
$ authClient 5qMix6Miv+O71+VYL95zk6U2wP79lKL3s1BFnd+a
$
$ 8BC90201EF55EE34F62DBA8FE8CF14DC.public/.private file created.
Here we keep the default values, including the safe generated password ({\f1\fs20 #weLHn5E.Qfe}). You should {\ul write down this password} in a safe place, because it will be required for any use of the private key, e.g. when signing or decrypting a message. If you forget about this password, there will be no way of accessing this private key any more - you have been warned!\line We will see @189@ how enabling the {\f1\fs20 ECC cheat} mode may help storing the generated {\f1\fs20 .private} key passwords in a {\f1\fs20 .cheat} encrypted local file using a {\f1\fs20 cheat.public} key, to safely recover a password, from a master {\f1\fs20 cheat.private} key and its associated password.
The last line contains the {\i identifier} (or serial) of the generated key. This hexadecimal value ({\f1\fs20 8BC90201EF55EE34F62DBA8FE8CF14DC}) will be used externally to identify the key, and internally (within other certificates) to map this particular key. Note that you do not need to type all the characters of the serial in the {\f1\fs20 ECC} tool: only the first characters are enough (e.g. {\f1\fs20 8BC9}), as soon as they identify one unique file in the current folder.
You can check the generated files in the current folder:
$>dir *.p*
$23/09/2016  13:46             2 320 8BC90201EF55EE34F62DBA8FE8CF14DC.private
$23/09/2016  13:46               544 8BC90201EF55EE34F62DBA8FE8CF14DC.public
The {\f1\fs20 .private} is some raw binary content, encrypted using the {\f1\fs20 #weLHn5E.Qfe} password.\line The {\f1\fs20 .public} file, on the contrary, is stored as a plain JSON object:
${
$ "Version": 1,
$ "Serial": "8BC90201EF55EE34F62DBA8FE8CF14DC",
$ "Issuer": "arbou",
$ "IssueDate": "2016-09-23",
$ "ValidityStart": "2016-09-23",
$ "ValidityEnd": "2017-09-23",
$ "AuthoritySerial": "8BC90201EF55EE34F62DBA8FE8CF14DC",
$ "AuthorityIssuer": "arbou",
$ "IsSelfSigned": true,
$ "Base64": "AQA1ADUAogGLyQIB71XuNPYtuo/ozxTcGrODgAAAAAAAAAAAAAAAAIvJAgHvVe409i26j+jPFNwas4OAAA
$AAAAAAAAAAAAAAAqjxQhWz5NjFLoWBsqvWyywne8ncNVSi/MmnVg+IZ4WknxoNvAia6oBBhC/tpo3zTjUQDssB8AId+/QR
$SF15RccuOy/j/ebeqX6qxANtZEZO3dT/sWBUjQy/CYIVQe5TSDZy5pQAAAAA"
$}
You can see all information stored in a {\f1\fs20 TECCCertificate} instance. The {\f1\fs20 "Base64"} field is in fact a raw serialization of the whole content, so its string value contains all information of a public certificate, e.g. in application settings.
We did not specify any authority at the first {\f1\fs20 Auth:} prompt. As a result, this key pair will be a {\i self-signed} certificate - see the {\f1\fs20 "IsSelfSigned": true} field in the above JSON, and that {\f1\fs20 "Serial"} and {\f1\fs20 "AuthoritySerial"} identifiers do match. We will use it as {\i root certificate} to create a certificate chain.
All further certificates will eventually be signed by this root authority.\line For instance:
$>ecc new
$Enter the first chars of the .private file name of the signing authority.
$Will create a self-signed certificate if left void.
$Auth: 8
$
$Will use: 8BC90201EF55EE34F62DBA8FE8CF14DC.private
$
$Enter the PassPhrase of this .private file.
$AuthPass: #weLHn5E.Qfe
$
$Enter the PassPhrase iteration rounds of this .private file.
$AuthRounds [60000] :
$
$Enter Issuer identifier text.
$Will be truncated to 15-20 ascii-7 chars.
$Issuer [arbou] : toto
$
$Enter the YYYY-MM-DD start date of its validity.
$0 will create a never-expiring certificate.
$Start [2016-09-23] : 0
$
$Enter a private PassPhrase for the new key (at least 8 chars long).
$Save this in a safe place: if you forget it, the key will be useless!
$NewPass [b3dEB+DW8BJd] :
$
$Corresponding TSynPersistentWithPassword.ComputePassword:
$ cIK5hkjDu5/98mwm
$
$Enter the PassPhrase iteration round for the new key (at least 1000).
$The higher, the safer, but will demand more computation time.
$NewRounds [60000] :
$
$ 03B8865C6B982A39E9EFB1DC1A95D227.public/.private file created.
As you can see, we entered just {\f1\fs20 8} for the first {\f1\fs20 Auth:} prompt, and the tool identified the single {\f1\fs20 8*.private} file in the current folder. Then we entered its associated {\f1\fs20 #weLHn5E.Qfe} password - any wrong password would have broken the generation. This authority will never expire by itself (we entered {\f1\fs20 0} as {\f1\fs20 Start:} prompt) - but since its root certificate has an expiration date, it will expire when the root expires.
Now we can see the two sets of keys:
$>dir *.p*
$23/09/2016  14:27             2 320 03B8865C6B982A39E9EFB1DC1A95D227.private
$23/09/2016  14:27               524 03B8865C6B982A39E9EFB1DC1A95D227.public
$23/09/2016  13:46             2 320 8BC90201EF55EE34F62DBA8FE8CF14DC.private
$23/09/2016  13:46               544 8BC90201EF55EE34F62DBA8FE8CF14DC.public
The newly created {\f1\fs20 .public} file contains:
${
$ "Version": 1,
$ "Serial": "03B8865C6B982A39E9EFB1DC1A95D227",
$ "Issuer": "toto",
$ "IssueDate": "2016-09-23",
$ "ValidityStart": "",
$ "ValidityEnd": "",
$ "AuthoritySerial": "8BC90201EF55EE34F62DBA8FE8CF14DC",
$ "AuthorityIssuer": "arbou",
$ "IsSelfSigned": false,
$ "Base64": "AQA1AAAAAAADuIZca5gqOenvsdwaldInhiGAAAAAAAAAAAAAAAAAAIvJAgHvVe409i26j+jPFNwas4OAAAA
$AAAAAAAAAAAAAA2rPT7XPmCH6xIt3+710FFYVAPBPWhcsR6uwYoyndrqNI7557iYC7qLrfKPgp6EmEENX4Vw0tEexu3O9SJ
$OAG/EHau0BtwoqZBNhJEiUyGqKa0ioMwasbKtjJBNMfX7EMIsnxn4AAAAA"
$}
You can recognize the expected values of {\f1\fs20 "Serial"}, {\f1\fs20 "AuthoritySerial"} and {\f1\fs20 "IsSelfSigned"} fields.
We could create a {\i certificates chain} of all available keys in the current folder, by running:
$>ecc chainall
$ chain.ca file created.
The {\f1\fs20 chain.ca} file is a JSON object, containing all public information of the whole certificates chain, with the {\f1\fs20 "PublicBase64"} JSON array ready to be copied and pasted in your applications settings or source, then used via the {\f1\fs20 TECCCertificateChain} class:
${
$ "PublicBase64":
$ [
$  "AQA1AAAAAA...",
$  "AQA1ADUAogG..."
$ ],
$ "Items":
$ [
$  {
$   "Version": 1,
$   "Serial": "03B8865C6B982A39E9EFB1DC1A95D227",
$   "Issuer": "toto",
$   "IssueDate": "2016-09-23",
$   "ValidityStart": "",
$   "ValidityEnd": "",
$   "AuthoritySerial": "8BC90201EF55EE34F62DBA8FE8CF14DC",
$   "AuthorityIssuer": "arbou",
$   "IsSelfSigned": false
$  },
$  {
$   "Version": 1,
$   "Serial": "8BC90201EF55EE34F62DBA8FE8CF14DC",
$   "Issuer": "arbou",
$   "IssueDate": "2016-09-23",
$   "ValidityStart": "2016-09-23",
$   "ValidityEnd": "2017-09-23",
$   "AuthoritySerial": "8BC90201EF55EE34F62DBA8FE8CF14DC",
$   "AuthorityIssuer": "arbou",
$   "IsSelfSigned": true
$  }
$ ],
$ "Count": 2,
$}
In the above sample, we cut down the {\f1\fs20 "PublicBase64"} values, to save some paper and trees. They map the content already shown in the {\f1\fs20 .public} JSON files. In fact, the same information is stored two times: once in {\f1\fs20 "PublicBase64"}, and another time in each individual properties ({\f1\fs20 "Version"}, {\f1\fs20 "Serial"}, {\f1\fs20 "Issuer"}...) of the {\f1\fs20 "Items"} items.
An easy way of keys management is to keep a safe mean of storage (e.g. a pair of USB pen-drives, with at least one kept in a physical vault), then put all your certificate chains in dedicated folders. All public keys - i.e. {\f1\fs20 *.public} and {\f1\fs20 chain.ca} files - are meant to be public, so could be spread away everywhere. Just keep an eye on your {\f1\fs20 .private} files, and their associated passwords. A hardware-secured drive may be an overkill, since the {\f1\fs20 .private} files are already encrypted and password-protected with state-of-the-art software protection, i.e. {\f1\fs20 AFSplit} anti-forensic diffusion and AES256-CFB encryption on a @*PBKDF2_HMAC_SHA256@ derived password, with a huge number of rounds (60000).
Remember that often, the weakest link of the security chain is between the chair and the keyboard, not within the computer. Do not reuse passwords between keys, and remember you have a "{\f1\fs20 rekey}" command available on the {\f1\fs20 ECC} tool, so that you can change a private key password, without changing its content, nor re-publish its associated {\f1\fs20 .public} key:
$>ecc rekey
$Enter the first chars of the .private certificate file name.
$Auth: 8
$
$Will use: 8BC90201EF55EE34F62DBA8FE8CF14DC.private
$
$Enter the PassPhrase of this .private file.
$AuthPass: #weLHn5E.Qfe
$
$Enter the PassPhrase iteration rounds of this .private file.
$AuthRounds [60000] :
$
$Enter a NEW private PassPhrase for the key (at least 8 chars long).
$Save this in a safe place: if you forget it, the key will be useless!
$NewPass [mPy3kjWHE@LK] :
$
$Corresponding TSynPersistentWithPassword.ComputePassword:
$ f+Gk8GGCqICA8GoJ
$
$Enter the NEW PassPhrase iteration round for the key (at least 1000).
$The higher, the safer, but will demand more computation time.
$NewRounds [60000] :
$
$ 8BC90201EF55EE34F62DBA8FE8CF14DC.private file created.
From now on, the root certificate will expect {\f1\fs20 mPy3kjWHE@LK} as keyphrase, for accessing its {\f1\fs20 .private} content. For instance (using only command line switches including the {\f1\fs20 -noprompt} option), you can now write:
$>ecc infopriv -auth 8B -pass mPy3kjWHE@LK -noprompt
${
$        "Version": 1,
$        "Serial": "8BC90201EF55EE34F62DBA8FE8CF14DC",
$        "Issuer": "arbou",
$        "IssueDate": "2016-09-23",
$        "ValidityStart": "2016-09-23",
$        "ValidityEnd": "2017-09-23",
$        "AuthoritySerial": "8BC90201EF55EE34F62DBA8FE8CF14DC",
$        "AuthorityIssuer": "arbou",
$        "IsSelfSigned": true,
$        "Base64": "AQA1ADUAogG...."
$}
Here, the {\f1\fs20 "Base64":} field only contains the public key information, not the private key content, which is kept secret and never serialized as JSON.
:  TECCCertificate and TECCCertificateSecret
As reference, here is how creating a new certificate is implemented in the {\f1\fs20 ECC} tool, and its {\f1\fs20 .private}/{\f1\fs20 .public} files generated, using {\f1\fs20 TECCCertificateSecret} class:
!function ECCCommandNew(const AuthPrivKey: TFileName;
!  const AuthPassword: RawUTF8; AuthPasswordRounds: integer;
!  const Issuer: RawUTF8; StartDate: TDateTime; ExpirationDays: integer;
!  const SavePassword: RawUTF8; SavePassordRounds, SplitFiles: integer): TFileName;
!var auth,new: TECCCertificateSecret;
!begin
!  if AuthPrivKey='' then
!    auth := nil else
!    auth := TECCCertificateSecret.CreateFromSecureFile(AuthPrivKey,AuthPassword,AuthPasswordRounds);
!  try
!    // generate pair
!    new := TECCCertificateSecret.CreateNew(auth,Issuer,ExpirationDays,StartDate);
!    try
!      // save private key as .private password-protected binary file
!      new.SaveToSecureFiles(SavePassword,'.',SplitFiles,64,SavePassordRounds);
!      // save public key as .public JSON file
!      result := ChangeFileExt(new.SaveToSecureFileName,ECCCERTIFICATEPUBLIC_FILEEXT);
!      ObjectToJSONFile(new,result);
!    finally
!      new.Free;
!    end;
!  finally
!    auth.Free;
!  end;
!end;
See the {\f1\fs20 SynEcc.pas} unit API reference, especially the {\f1\fs20 TECCCertificateChain} and {\f1\fs20 TECCCertificateChainFile} classes, which allow to store a certificate chain as JSON files or as a JSON array of base-64 encoded strings in your settings, using these constructors:
!    constructor CreateFromJson(const json: RawUTF8);
!    constructor CreateFromArray(const values: TRawUTF8DynArray);
You can use the "{\f1\fs20 source}" command of the ECC tool to generate some pascal constant source code, containing an encrypted private key, ready to be embedded to your executable. For instance:
$>ecc source -auth 8 -pass mPy3kjWHE@LK -const MY_PRIV -noprompt
$Will use: 8BC90201EF55EE34F62DBA8FE8CF14DC.private
$
$ 8BC90201EF55EE34F62DBA8FE8CF14DC.private.inc file created.
When you look at the {\f1\fs20 .private.inc} generated file, you can directly use it in your source code, via copy and paste:
!const
!  MY_PRIV: array[0..255] of byte = (
!    $C3,$31,$17,$48,$35,$B6,$30,$AA,$87,$ED,$AE,$DF,$84,$29,$AA,$85,
!    $8D,$BF,$A0,$05,$92,$7F,$6C,$47,$66,$D7,$23,$B3,$5B,$4C,$42,$97,
!    $5B,$07,$73,$3B,$FE,$FA,$BE,$A7,$96,$9B,$F9,$1D,$84,$CC,$6E,$F0,
!    $C9,$A1,$A7,$2A,$24,$8D,$4A,$B3,$F3,$B3,$89,$52,$70,$46,$83,$84,
!    $5D,$FD,$E7,$E9,$C3,$D9,$DC,$07,$C1,$FF,$82,$F8,$78,$45,$BF,$18,
!    $CA,$F7,$EE,$8E,$A3,$42,$0D,$0B,$35,$2F,$20,$4A,$65,$82,$4A,$78,
!    $C4,$41,$19,$0E,$98,$77,$7D,$81,$58,$DB,$04,$C9,$52,$2E,$5F,$07,
!    $CF,$44,$34,$93,$1B,$FD,$00,$38,$E0,$E7,$DC,$3A,$AC,$CB,$14,$73,
!    $B2,$E0,$13,$BE,$84,$79,$F7,$55,$8A,$12,$4F,$9A,$09,$97,$CC,$9B,
!    $8E,$7C,$04,$92,$93,$24,$73,$50,$41,$B3,$92,$54,$D7,$66,$05,$4A,
!    $3E,$4F,$D4,$1B,$94,$71,$AA,$04,$29,$42,$B3,$57,$B0,$F3,$24,$74,
!    $19,$8E,$BA,$52,$FA,$D6,$56,$99,$7B,$73,$1B,$D0,$8B,$3A,$95,$AB,
!    $94,$63,$C2,$C0,$78,$05,$9C,$8B,$85,$B7,$A1,$E3,$ED,$93,$27,$18,
!    $F6,$DD,$87,$D7,$E9,$35,$74,$01,$2E,$35,$DF,$1A,$6E,$FA,$4A,$3F,
!    $E3,$70,$19,$A3,$D7,$E3,$39,$37,$59,$15,$43,$B2,$F4,$36,$B5,$64,
!    $D6,$BF,$75,$12,$6B,$C5,$89,$95,$2D,$E5,$70,$64,$EB,$70,$98,$9D);
!  MY_PRIV_LEN = SizeOf(MY_PRIV);
!  MY_PRIV_ROUNDS = 100;
!  MY_PRIV_SERIAL = '8BC90201EF55EE34F62DBA8FE8CF14DC';
!  MY_PRIV_PASS = 'Oute?/I#JSxL0VcLeR/HY(yr';
!  MY_PRIV_CYPH = '2R3VSFXfikiNaKTM3MqbTlw+PiMBwshk';
With these classes, you have everything needed to implement your own private and secure {\f1\fs20 @*PKI@} logic in your client/server applications.
:  File Signature
Starting from those two text files:
$>dir test*.*
$19/09/2016  21:46            94 161 test1.txt
$05/09/2016  10:37            72 209 test2.txt
In order to sign {\f1\fs20 test1.txt}, as proposed in @%%AsymmSign@, we will run the following command:
$>ecc sign -file test1.txt
$Enter the first chars of the .private file name of the signing authority.
$Auth: 8B
$
$Will use: 8BC90201EF55EE34F62DBA8FE8CF14DC.private
$
$Enter the PassPhrase of this .private file.
$Pass: mPy3kjWHE@LK
$
$Enter the PassPhrase iteration rounds of this .private file.
$Rounds [60000] :
$
$ test1.txt.sign file created.
As you can see, a new {\f1\fs20 .sign} file appeared:
$>dir test*.*
$19/09/2016  21:46            94 161 test1.txt
$24/09/2016  16:24               369 test1.txt.sign
$05/09/2016  10:37            72 209 test2.txt
It is a simple JSON object, with some information about the associated {\f1\fs20 test1.txt} file:
${
$        "meta": {
$                "name": "test1.txt",
$                "date": "2016-09-19T21:46:24"
$        },
$        "size": 94161,
$        "md5": "e80f2bf959c943e240f2c1f5efcf1e89",
$        "sha256": "6bc2d25e9cc93201914e7c6588624696778de80c6f63a590262ccf610310ea0e",
$        "sign": "AQA2AIvJAgHvVe409i26j+jPFNwas4OAAAAAAAAAA...."
$}
In additional to some general information (name, date, size), you have unsigned hashes ({\f1\fs20 "md5"} and {\f1\fs20 "sha256"}), and an ECC digital signature, stored as a base-64 encoded string in the {\f1\fs20 "sign":} field. This signature has been computed using the {\f1\fs20 8BC90201EF55EE34F62DBA8FE8CF14DC.private} key, and the @*SHA256@ hash of the {\f1\fs20 test1.txt} file content. Note that you can add whatever JSON field you need to any {\f1\fs20 .sign} file, especially in the {\f1\fs20 "meta":} nested object, as soon as you don't modify the {\f1\fs20 size/md5/sha256/sign} values.
To verify the file, ensure that both {\f1\fs20 test1.txt} and {\f1\fs20 test1.txt.sign} files are in the current directory, then run:
$>ecc verify -file test1.txt
$ test1.txt file verified as valid self signed.
Since the {\f1\fs20 8BC90201EF55EE34F62DBA8FE8CF14DC.private} key has been signed using itself as authority, it is reported as "{\f1\fs20 valid self signed}". A signature verified against a certificate itself issued from another authority would have returned "{\f1\fs20 valid signed}".
Now if you modify {\f1\fs20 test1.txt}, e.g. changing one character, the verification will fail:
$>ecc verify -file test1.txt
$ test1.txt file verification failure: invalid signature (9).
Don't forget to fix the test1.txt content back, since we will use it now as encryption source.
To check that you reverted to the original file content, run:
$>ecc verify -file test1.txt
$ test1.txt file verified as valid self signed.
:  Signing in Code
From the source code point of view, you can easily add asymmetric digital signatures in your project using the {\f1\fs20 TECCCertificateSecret.SignFile} method, or working with memory buffer instead of files thanks to {\f1\fs20 TECCCertificateSecret.SignToBase64} overloaded methods.
As reference, here is how the signing is implemented in the {\f1\fs20 ECC} tool:
!function ECCCommandSignFile(const FileToSign, AuthPrivKey: TFileName;
!  const AuthPassword: RawUTF8; AuthPasswordRounds: integer): TFileName;
!var auth: TECCCertificateSecret;
!begin
!  auth := TECCCertificateSecret.CreateFromSecureFile(AuthPrivKey,AuthPassword,AuthPasswordRounds);
!  try
!    result := auth.SignFile(FileToSign,[]);
!  finally
!    auth.Free;
!  end;
!end;
Verification can be done via the dedicated {\f1\fs20 TECCSignatureCertified} class:
!function ECCCommandVerifyFile(const FileToVerify, AuthPubKey: TFileName;
!  const AuthBase64: RawUTF8): TECCValidity;
!var content: RawByteString;
!    auth: TECCCertificate;
!    cert: TECCSignatureCertified;
!begin
!  content := StringFromFile(FileToVerify);
!  if content='' then
!    raise EECCException.CreateUTF8('File not found: %',[FileToVerify]);
!  cert := TECCSignatureCertified.CreateFromFile(FileToVerify);
!  try
!    if not cert.Check then begin
!      result := ecvInvalidSignature;
!      exit;
!    end;
!    auth := TECCCertificate.Create;
!    try
!      if auth.FromAuth(AuthPubKey,AuthBase64,cert.AuthoritySerial) then
!        result := cert.Verify(auth,pointer(content),length(content)) else
!        result := ecvUnknownAuthority;
!    finally
!      auth.Free;
!    end;
!  finally
!    cert.Free;
!  end;
!end;
Here, the signing authority is supplied as a single {\f1\fs20 .public} local file, loaded in a {\f1\fs20 TECCCertificate} instance, but your projects may use {\f1\fs20 TECCCertificateChain} for a full {\f1\fs20 @*PKI@} authority chain.
:190  File Encryption
In order to encrypt out both test files, as proposed in @%%AsymmEncrypt@, we will run the following commands:
$>ecc crypt -file test1.txt -auth 03 -saltpass monsecret -noprompt
$Will use: 03B8865C6B982A39E9EFB1DC1A95D227.public
$
$ test1.txt.synecc file created.
$
$>ecc crypt -file test2.txt -auth 03 -saltpass monsecret2 -noprompt
$Will use: 03B8865C6B982A39E9EFB1DC1A95D227.public
$
$ test2.txt.synecc file created.
As we can see, two new {\f1\fs20 .synecc} encrypted files have been computed:
$>dir test*
$24/09/2016  16:36            94 161 test1.txt
$24/09/2016  16:24               369 test1.txt.sign
$24/09/2016  17:13            22 436 test1.txt.synecc
$05/09/2016  10:37            72 209 test2.txt
$24/09/2016  17:13            15 220 test2.txt.synecc
You may notice that the {\f1\fs20 .synecc} files are smaller than the original {\f1\fs20 .txt} files... in fact, {\f1\fs20 SynEcc} did recognize that the plain content was easily compressible, then applied {\f1\fs20 @*SynLZ@} compression on it, before the encryption step.
If we ask for information about the {\f1\fs20 test1.txt.synecc} file:
$>ecc infocrypt -file test1.txt.synecc
${
$        "Date": "2016-09-24",
$        "Size": 94161,
$        "Recipient": "toto",
$        "RecipientSerial": "03B8865C6B982A39E9EFB1DC1A95D227",
$        "FileTime": "2016-09-24T16:36:59",
$        "Algorithm": "ecaPBKDF2_HMAC_SHA256_AES256_CFB_SYNLZ",
$        "RandomPublicKey": "038C73D421D9A7F2F7FFC0F3EF62C79897D13064EF2D6FAA0DB0F246CD9B173B90",
$        "HMAC": "c6bf56792e58c68c45ebdff1fbd6a3b2e4d3e95e522f41eba2ecab41fd53183b",
$        "Signature": {
$                "Version": 1,
$                "Date": "2016-09-24",
$                "AuthoritySerial": "8BC90201EF55EE34F62DBA8FE8CF14DC",
$                "AuthorityIssuer": "arbou",
$                "ECDA": "2AwyyNYAcfSyJW+5BzvksbSdXcOUbYNqm...."
$        }
$}
We can see the information stored in the file header, including the recipient name and {\f1\fs20 .publickey} identifier, and also the {\f1\fs20 "PBKDF2_HMAC_SHA256_AES256_CFB_SYNLZ"} algorithm, which indeed includes {\f1\fs20 _SYNLZ} compression. Other algorithms are available (with diverse {\f1\fs20 @*AES@} chaining modes), and some new methods may be added in the future.
The {\f1\fs20 ecc crypt} command did also include the digital signature available in the {\f1\fs20 test1.txt.sign} file in the current folder - so was in fact following @%%AsymmSignEncrypt@ - whereas {\f1\fs20 test2.txt.synecc} does not have any embedded signature, since there was no {\f1\fs20 test2.txt.sign} file available at encryption time:
$>ecc infocrypt -file test2.txt.synecc
${
$        "Date": "2016-09-24",
$        "Size": 72209,
$        "Recipient": "toto",
$        "RecipientSerial": "03B8865C6B982A39E9EFB1DC1A95D227",
$        "FileTime": "2016-09-05T10:37:50",
$        "Algorithm": "ecaPBKDF2_HMAC_SHA256_AES256_CFB_SYNLZ",
$        "RandomPublicKey": "03914AA67BCC92F78A9D670B2209587E3C97A3E08FD39A38B459558511F88F7651",
$        "HMAC": "07ddc90f7695fff8ca0683be98f7b1043e13a7bceb79f0d6a929069c5d9767d1",
$        "Signature": null
$}
As you can see, encryption is defined by its {\f1\fs20 "Algorithm":} field, and uses two additional properties:
- {\f1\fs20 "RandomPublicKey"} which contains a genuine key generated by {\f1\fs20 ecc crypt}, allowing {\i @**perfect forward secrecy@}, meaning that a shared secret key is computed for every encryption: if someone achieves to break the AES256-CFB secret key used to encrypt a particular {\f1\fs20 .synecc} file (e.g. spending lots of money in brute force search), this secret key won't be reusable for any other file: each {\f1\fs20 "RandomPublicKey"} value above is indeed unique for each {\f1\fs20 .synecc} file;
- {\f1\fs20 "HMAC":} which uses a safe way of message authentication - known as {\i keyed-hash message authentication code} (@*HMAC@) - stronger than the hashing algorithm it is based on, i.e. @*SHA256@ in our case.
In practice, {\f1\fs20 SynEcc} implements state-of-the-art {\i Elliptic Curve Integrated Encryption Scheme} ({\f1\fs20 @*ECIES@}) using @*PBKDF2_HMAC_SHA256@ as key derivation function, AES256-CFB as symmetric encryption scheme, and @*HMAC-SHA256@ algorithm for message authentication.\line  See @https://en.wikipedia.org/wiki/Integrated_Encryption_Scheme
{\f1\fs20 ECIES} provides semantic security against an adversary who is allowed to use chosen-plaintext and chosen-ciphertext attacks. In addition to the expected genuine secret and message authentication in {\f1\fs20 "RandomPublicKey"} and {\f1\fs20 "HMAC"} properties, {\f1\fs20 SynEcc} implementation allows to customize the default "salt" value, to add a password protection for each {\f1\fs20 .synecc} encrypted file.
Decryption is pretty straightforward:
$>ecc decrypt -file test1.txt.synecc
$Enter the name of the decrypted file
$Out [test1.txt.2] :
$
$Enter the PassPhrase of the associated .private file.
$AuthPass: b3dEB+DW8BJd
$
$Enter the PassPhrase iteration rounds of this .private file.
$AuthRounds [60000] :
$
$Enter the optional PassPhrase to be used for decryption.
$SaltPass [salt] : monsecret
$
$Enter the PassPhrase iteration rounds.
$SaltRounds [60000] :
$
$ test1.txt.2 file verified as valid self signed.
$ test1.txt.synecc file decrypted with signature.
$ test1.txt.2 file created.
To decrypt the second file in a single step, and no console interaction:
$>ecc decrypt -file test2.txt.synecc -authpass b3dEB+DW8BJd -saltpass monsecret2 -noprompt
$ test2.txt.synecc file decrypted.
$ test2.txt.2 file created.
As expected, the second file didn't contain any digital signature, so there is no "{\f1\fs20 test2.txt.2 file verified as valid self signed.}" message.
The decrypted files are available in the current folder:
$> dir test*
$24/09/2016  16:36            94 161 test1.txt
$24/09/2016  16:36            94 161 test1.txt.2
$24/09/2016  16:24               369 test1.txt.sign
$24/09/2016  17:13            22 436 test1.txt.synecc
$05/09/2016  10:37            72 209 test2.txt
$05/09/2016  10:37            72 209 test2.txt.2
$24/09/2016  17:13            15 220 test2.txt.synecc
The {\f1\fs20 *.2} decrypted files have the expect size (and content), after decompression. Even the file timestamp has been set to match the original.
:189  Private Keys Passwords Cheat Mode
In order to follow best practice, our {\f1\fs20 .private} key files are always protected by a password. A random value with enough length and entropy is always proposed by the {\f1\fs20 ECC} tool when a key pair is generated, and could be used directly. It is always preferred to trust a computer to create true randomness (and {\f1\fs20 SynCrypto.pas}'s secure {\f1\fs20 TAESPRNG} was designed to be the best possible seed, using hardware entropy if available), than using our human brain, which could be defeated by dictionary-based password attacks. Brute force cracking would be almost impossible, since {\f1\fs20 @**PBKDF2_HMAC_SHA256@} Password-Based Key Derivation Function with 60,000 rounds is used, so rainbow tables (i.e. pre-computed passwords list) will be inoperative, and each password trial would take more time than with a regular Key Derivation Function.
The issue with strong passwords is that they are difficult to remember. If you use not pure random passwords, but some easier to remember values with good entropy, you may try some tools like @https://xkpasswd.net/s which returns values like {\f1\fs20 $$19*wrong*DRIVE*read*61$$}. But even then, you will be able to remember only a dozen of such passwords. In a typical public key infrastructure, you may create hundredths of keys, so remembering all passwords is no option for an average human being as you and me.
At the end, you end up with using a tool to store all your passwords (last trend is to use an online service with browser integration), or - admit it - store them in an {\f1\fs20 Excel} document protected by a password. Most IT people - and even security specialists - end with using such a mean of storage, just because they need it.\line The weaknesses of such solutions can be listed:
- How could we trust closed source software and third-party online services?
- Even open source like @http://keepass.info/help/base/security.html may appear weak (no PBKDF, no AFSplit, managed C#, SHA as PRNG);
- The storage is as safe as the "master password" is safe;
- If the "master password" is compromised, all your passwords are published;
- You need to know the master password to add a new item to the store.
The {\f1\fs20 ECC} tool is able to work in "cheat mode", storing all {\f1\fs20 .private} key files generated passwords in an associated {\f1\fs20 .cheat} local file, encrypted using a {\f1\fs20 cheat.public} key.\line As a result:
- Each key pair will have its own associated {\f1\fs20 .cheat} file, so you only unleash one key at a time;
- The {\f1\fs20 .cheat} file content is meaningless without the {\f1\fs20 cheat.private} key and its master password, so you can manage and store them together with your {\f1\fs20 .private} files;
- Only the {\f1\fs20 cheat.public} key is needed when creating a key pair, so you won't leak your master password, and even could generate keys in an automated way, on a distant server;
- The {\f1\fs20 cheat.private} key will be safely stored in a separated place, only needed when you need to recover a password;
- It uses strong @190@, with proven PBKDF, AFSplit, AES-PRNG, and ECDH/ECIES algorithms.
By default, no {\f1\fs20 .cheat} files are created. You need to explicitly initialize the "cheat mode", by creating master {\f1\fs20 cheat.public} and {\f1\fs20 cheat.private} key files:
$ >ecc cheatinit
$Enter Issuer identifier text of the master cheat keys.
$Will be truncated to 15-20 ascii-7 chars.
$Issuer [arbou] :
$
$Enter a private PassPhrase for the master cheat.private key (at least 8 chars).
$Save this in a safe place: if you forget it, the key will be useless!
$NewPass [uQHH*am39LLj] : verysafelongpassword
$
$Enter iteration rounds for the mastercheat.private key (at least 100000).
$NewRounds [100000] :
$
$ cheat.public/.private file created.
$
As you can see, the default number of PBKDF rounds is high (100000), and local files have been created:
$>dir cheat.*
$
$18/10/2016  11:12             4 368 cheat.private
$18/10/2016  11:12               568 cheat.public
$
Now we will create a new key pair (in a single command line, with no console interaction):
$>ecc new -newpass NewKeyP@ssw0rd -noprompt
$
$Corresponding TSynPersistentWithPassword.ComputePassword:
$ encryption HeOyjDUAsOhvLZkMA0Y=
$ authMutual lO0mv+8VpoFrrFfbBFilNppn1WumaIL+AN3JXEUUpCY=
$ authServer lO0nv+8VpoFrrFfbBFilNppn1WumaIL+AN3JXEUUpCY=
$ authClient lO0kv+8VpoFrrFfbBFilNppn1WumaIL+AN3JXEUUpCY=
$
$ D1045FCBAA1382EE44ED2C212596E9E1.public/.private file created.
$
An associated {\f1\fs20 .cheat} file has been created:
$>dir D10*
$
$18/10/2016  11:15             1 668 D1045FCBAA1382EE44ED2C212596E9E1.cheat
$18/10/2016  11:15             2 320 D1045FCBAA1382EE44ED2C212596E9E1.private
$18/10/2016  11:15               588 D1045FCBAA1382EE44ED2C212596E9E1.public
$
Imagine you forgot about the {\f1\fs20 NewKeyP@ssw0rd} value. You could use the following command to retrieve it:
$>ecc cheat
$
$Enter the first chars of the .private certificate file name.
$Auth: D10
$
$Will use: D1045FCBAA1382EE44ED2C212596E9E1.private
$
$Enter the PassPhrase of the master cheat.private file.
$AuthPass: verysafelongpassword
$
$Enter the PassPhrase iteration rounds of the cheat.private file.
$AuthRounds [100000] :
$
${
$        "pass": "NewKeyP@ssw0rd",
$        "rounds": 60000
$}
$Corresponding TSynPersistentWithPassword.ComputePassword:
$ encryption HeOyjDUAsOhvLZkMA0Y=
$ authMutual lO0mv+8VpoFrrFfbBFilNppn1WumaIL+AN3JXEUUpCY=
$ authServer lO0nv+8VpoFrrFfbBFilNppn1WumaIL+AN3JXEUUpCY=
$ authClient lO0kv+8VpoFrrFfbBFilNppn1WumaIL+AN3JXEUUpCY=
$
If your {\f1\fs20 .private} key does not have its associated {\f1\fs20 .cheat} file, you won't be able to recover your password:
$>ecc cheat
$
$Enter the first chars of the .private certificate file name.
$Auth: 8BC9
$
$Will use: 8BC90201EF55EE34F62DBA8FE8CF14DC.private
$
$Enter the PassPhrase of the master cheat.private file.
$AuthPass: verysafelongpassword
$
$Enter the PassPhrase iteration rounds of the cheat.private file.
$AuthRounds [100000] :
$
$Fatal exception EECCException raised with message:
$ Unknown file 8BC90201EF55EE34F62DBA8FE8CF14DC.cheat
$
In practice, this "cheat mode" will help you implement a safe public key infrastructure of any size. It will be as secure as the main {\f1\fs20 cheat.private} key file and its associated password remain hidden and only wisely spread, of course. Don't forget to use the {\f1\fs20 ecc rekey} command on a regular basis, so that you change the master password of {\f1\fs20 cheat.private}. The main benefit of this implementation is that for all key generation process, only the {\f1\fs20 cheat.public} key file is needed.
:  Encryption in Code
From the source code point of view, you can easily add asymmetric encryption in your project using {\f1\fs20 TECCCertificate.EncryptFile} and {\f1\fs20 TECCCertificateSecret.DecryptFile} methods, even working with memory buffers, thanks to {\f1\fs20 TECCCertificate.Encrypt} and {\f1\fs20 TECCCertificateSecret.Decrypt} methods.
As reference, here is how the encryption is implemented in the {\f1\fs20 ECC} tool:
!procedure ECCCommandCryptFile(const FileToCrypt, DestFile, AuthPubKey: TFileName;
!  const AuthBase64, AuthSerial, Password: RawUTF8; PasswordRounds: integer; Algo: TECIESAlgo);
!var content: RawByteString;
!    auth: TECCCertificate;
!begin
!  content := StringFromFile(FileToCrypt);
!  if content='' then
!    raise EECCException.CreateUTF8('File not found: %',[FileToCrypt]);
!  auth := TECCCertificate.Create;
!  try
!    if auth.FromAuth(AuthPubKey,AuthBase64,AuthSerial) then begin
!      auth.EncryptFile(FileToCrypt,DestFile,Password,PasswordRounds,Algo,true);
!    end;
!  finally
!    auth.Free;
!    FillZero(content);
!  end;
!end;
You may note here the use of {\f1\fs20 FillZero()} in the {\f1\fs20 finally} block of the function, which is a common - and strongly encouraged - way of protecting your sensitive data from remaining in RAM, after use. Both {\f1\fs20 SynCrypto.pas} and {\f1\fs20 SynEcc.pas} code has been checked to follow similar safety patterns, and not leave any sensitive information in the program stack or heap.
:193 Application Locking
A common feature request for professional software is to prevent abuse of published applications. For licensing or security reasons, you may be requested to "lock" the execution of programs, maybe tools or services.
{\i mORMot} can use @*Asymmetric@ Cryptography to ensure that only allowed users could run some executables, optionally with dedicated settings, on a given computer. The framework offers the first brick, on which you should build upon your dedicated system.
The {\f1\fs20 dddInfraApps.pas} unit publishes the following {\f1\fs20 @*ECCAuthorize@} function and type:
!type
!  TECCAuthorize = (eaSuccess, eaInvalidSecret, eaMissingUnlockFile,
!    eaInvalidUnlockFile, eaInvalidJson);
!
!function ECCAuthorize(aContent: TObject; aSecretDays: integer; const aSecretPass,
!  aDPAPI, aDecryptSalt, aAppLockPublic64: RawUTF8; const aSearchFolder: TFileName = '';
!  aSecretInfo: PECCCertificateSigned = nil; aLocalFile: PFileName = nil): TECCAuthorize;
This function will use several asymmetric key sets:
- A {\i main key set}, named e.g. {\f1\fs20 applock.public} and {\f1\fs20 applock.private}, shared for all users of the system;
- Several {\i user-specific key sets}, named e.g. {\f1\fs20 user@host.public} and {\f1\fs20 user@host.secret}, one for each {\f1\fs20 user} and associated computer {\f1\fs20 host} name.
When the {\f1\fs20 ECCAuthorize} function is executed, it will search for a local {\f1\fs20 user@host.unlock} file, named after the current logged user and the computer host name. Of course, the first time the application is launched for this user, there will be no such file. It will create two local {\f1\fs20 user@host.public} and {\f1\fs20 user@host.secret} files and return {\f1\fs20 eaMissingUnlockFile}.
The {\i main key set} will be used to digitally {\i sign} the {\f1\fs20 unlock} file:
- {\f1\fs20 applock.public} will be supplied as plain base64-encoded {\f1\fs20 aAppLockPublic64} text parameter in the executables - for safety, you should ensure its value is note replaced by a forged one by an attacker: the executable should be signed, or at least the constant value should be checked with a CRC for its content during the program execution;
- On the contrary, {\f1\fs20 applock.private} will be kept secret - with its associated secret password.
{\i User-specific key sets} will be used to {\i encrypt} the {\f1\fs20 unlock} file:
- The {\f1\fs20 user@host.secret} file contains in fact a genuine private key, encrypted using {\f1\fs20 CryptDataForCurrentUser} (i.e. {\f1\fs20 DPAPI} under {\i Windows}) for the specific computer and user: this will avoid {\f1\fs20 user@host.unlock} reuse on another computer, even if the user and host names are identical, and the {\f1\fs20 user@host.secret} file is copied. This file should remain local, and doesn't need to be transmitted.
- The {\f1\fs20 user@host.public} file will be sent to the product support team, e.g. by email - but you may setup an automated server, if needed. The support team will create a {\f1\fs20 user@host.unlock} matching this {\f1\fs20 user@host.public} key, which will unlock the application for the given user.
On the support team side, a {\f1\fs20 user@host.json} file is created for the given user, and will contain the @*JSON@ serialization of the {\f1\fs20 aContent: TObject} parameter of the {\f1\fs20 ECCAuthorize} function. This object may contain any published properties, matching the security expectations for this user, e.g. the available features or resource access.
:  From the User perspective
The resulting process is therefore the following:
\graph ECCAuthorizeWorkflow1 Application Unlocking via Asymmetric Cryptography
subgraph cluster_0 {
label="PC1 Computer";
\Application\user1@pc1.secret
\Application\user1@pc1.public
\Application\user1@pc1.unlock
\Application\applock.public
}
subgraph cluster_3 {
label="Support Team";
\user1@pc1.public\ user1@pc1.public\email
\user1@pc1.json\ user1@pc1.unlock\encrypt
\applock.private\ user1@pc1.unlock\sign
\ user1@pc1.public\ user1@pc1.unlock
\ user1@pc1.unlock\user1@pc1.unlock\email
}
\
In short, every user/computer combination will have its own set of {\f1\fs20 public/secret/unlock} files.
- In practice, {\f1\fs20 applock.public} could be hardcoded as plain base64-encoded {\f1\fs20 aAppLockPublic64} constant {\f1\fs20 string} in the Application code - of course, the executable should be signed with a proper authority, to ensure this constant is not replaced by a fake value;
- The location of those local {\f1\fs20 user@host.*} files is by default the executable folder, but may be specified via the {\f1\fs20 aSearchFolder} parameter - especially if this folder is read-only (e.g. due to Windows UAC), or if you use some custom GUI for the user interactivity;
- The {\f1\fs20 user@host.json} will be signed using {\f1\fs20 applock.private} secret key, to testify that the resulting {\f1\fs20 user@host.unlock} file was indeed provided by the Support Team;
- The {\f1\fs20 user@host.json} will be encrypted using the {\f1\fs20 user@host.public} key received by email, so will be specific to a single user/computer combination.
If two users share the application on the very same computer, another set of files will appear:
\graph ECCAuthorizeWorkflow2 Application Unlocking on Two Computers
subgraph cluster_0 {
label="PC1 Computer";
\Application\user1@pc1.secret
\Application\user1@pc1.public
\Application\user1@pc1.unlock
\Application\applock.public
}
subgraph cluster_1 {
label="PC2 Computer";
\ Application\user2@pc2.secret
\ Application\user2@pc2.public
\ Application\user2@pc2.unlock
\ Application\ applock.public
}
subgraph cluster_3 {
label="Support Team";
\user1@pc1.public\ user1@pc1.public\email
\ user1@pc1.public\ user1@pc1.unlock\encrypt
\user1@pc1.json\ user1@pc1.unlock
\applock.private\ user1@pc1.unlock\sign
\user2@pc2.public\ user2@pc2.public\email
\user2@pc2.json\ user2@pc2.unlock\encrypt
\applock.private\ user2@pc2.unlock\sign
\ user1@pc1.unlock\user1@pc1.unlock\email
\ user2@pc2.public\ user2@pc2.unlock
\ user2@pc2.unlock\user2@pc2.unlock\email
}
\
Several users on the same computer will be handled as such:
\graph ECCAuthorizeWorkflow3 Application Unlocking for Two Users
subgraph cluster_0 {
label="PC1 Computer";
\Application\applock.public
\Application\user2@pc1.public
\Application\user2@pc1.unlock
\Application\user2@pc1.secret
\Application\user1@pc1.public
\Application\user1@pc1.unlock
\Application\user1@pc1.secret
}
subgraph cluster_3 {
label="Support Team";
\user1@pc1.json\ user1@pc1.unlock\encrypt
\user1@pc1.public\ user1@pc1.public\email
\ user1@pc1.public\ user1@pc1.unlock
\ user1@pc1.unlock\user1@pc1.unlock\email
\applock.private\ user1@pc1.unlock\sign
\user2@pc1.json\ user2@pc1.unlock
\user2@pc1.public\ user2@pc1.public\email
\ user2@pc1.public\ user2@pc1.unlock\encrypt
\ user2@pc1.unlock\user2@pc1.unlock
\applock.private\ user2@pc1.unlock\sign
}
\
From the User point of view, he/she will transmit its {\f1\fs20 user@host.public} file, then receives a corresponding {\f1\fs20 user@host.unlock} file, which will unlock the application. Pretty easy to understand - even if some complex asymmetric encryption is involved behind the scene.
:  From the Support Team perspective
The Support Team will maintain a list of {\f1\fs20 user@host.public} and {\f1\fs20 user@host.json} files, one per user/computer. Both files have small JSON content, so may be stored in a dedicated folder of the project source code repository - or in a dedicated repository. The use of a source code repository allows to track user management information between several support people, including history and audit trail of this sensitive information. For safety, the {\f1\fs20 applock.private} file may not be archived in the source code repository, but copied on purpose on each support people's (or developer's) computer. A separated, and dedicated computer, may be used, for additional safety.
In fact, even developers may define their own set of {\f1\fs20 .unlock} files. For local test builds, they may use their own {\f1\fs20 applock.public} and {\f1\fs20 applock.private} key pairs, diverse from the main content.
The content of each {\f1\fs20 user@host.json} may be easily derivated from a set of reference {\f1\fs20 .json} files, acting like templates of group of users. Or an existing file may be used as source for a new user. The ability to use JSON and a text editor, with customizable object and arrays fields, allows any needed kind of licensing or security scope, depending on the application.\line Since the {\f1\fs20 user@host.json} is a serialized {\f1\fs20 aContent: TObject}, you can define enumerates properties, or even schema-less structures as {\f1\fs20 @*TDocVariant@} - see @80@ - to refine the authorization scope.
The {\f1\fs20 user@host.json} file is encrypted using the genuine {\f1\fs20 user@host.public} key, and its associated {\f1\fs20 user@host.secret} is strongly encrypted for the given PC and logged user: therefore, only the application is able to decipher the {\f1\fs20 user@host.unlock} content. You can let those files be transmitted via an unsafe mean of transport, e.g. plain email, with no compromising risk. Last but not least, passwords or IP addresses can be safely stored in its content, as part of the security policy of your project.
In practice, the team may use a {\f1\fs20 unlock.bat} file running the ECC tool over secret {\f1\fs20 applock.private} keys, containing the secret:
$ @echo off
$ echo Usage:  unlock user@host
$ echo.
$ ecc sign -file %1.json -auth applock -pass applockprivatepassword -rounds 60000
$ ecc crypt -file %1.json -out %1.unlock -auth %1 -saltpass decryptsalt -saltrounds 10000
$ del %1.json.sign
For safety, you may not include the {\f1\fs20 -pass applockprivatepassword} value in this {\f1\fs20 unlock.bat} file. Removing this {\f1\fs20 -pass} command-line switch will let the {\f1\fs20 ecc} tool prompt for the password secret key on the console:
$ ecc sign -file %1.json -auth applock -rounds 60000
Also note that you can use the {\f1\fs20 ecc rekey} command to customize the password of a given {\f1\fs20 applock.private} file: each support team member may have his/her custom password to run the {\i sign-then-encrypt} process.
Of course, if you need to create a lot of {\f1\fs20 .unlock} files, you may want to automate this process, e.g. in a server or a GUI tool, using {\f1\fs20 SynEcc.pas} classes.
:  Benefits of Asymmetric Encryption for License management
In most licensing systems, the weak point is the transmission of the licensing file. Thanks to Asymmetric Encryption, both {\f1\fs20 user@host.public} and {\f1\fs20 user@host.unlock} files can be transmitted as plain emails, without any possibility of compromising.
The {\f1\fs20 applock.private} secret key and its associated password are used to digitally sign (using ECDSA) the plain content of the {\f1\fs20 user@host.unlock} file. This {\i sign-then-encrypt} pattern will ensure that only your support team will be able to generate the proper {\f1\fs20 .unlock} files for a given application. The {\f1\fs20 applock.private/public} keys could have their own deprecation date.
As we have seen, the {\f1\fs20 user@host.unlock} file is encrypted, so you can use it to transmit sensitive information. Its associated {\f1\fs20 user@host.secret} key has been generated locally with an expiration date - see the {\f1\fs20 aSecretDays} parameter of the {\f1\fs20 ECCAuthorize} function. It will ensure that the registering process should be performed regularly, if the licensing or security policy expect it.
Of course, any such system is as weak as its weakest point. In particular, under Windows the executable should be digitally signed (as any professional software). You could also ensure that the {\f1\fs20 aAppLockPublic64} public key has not been replaced by a fake value forged by an attacker - e.g. by checking its value by computing its CRC in several places of your application:
! if crc32($1239438,pointer(AppLock64),length(AppLock64))<>$ae293c10 then Close;
The security of this system does not rely on code obfuscation, but on proven safety of asymmetric encryption. Even if the executable is modified in-place to by-pass the license check, the fact that the application expects some additional information to be provided within the {\f1\fs20 user@host.unlock} file will make it much more difficult to hack.\line As always with Open Source, any feedback is welcome, in order to enhance the safety of this system. The fact that the code is available - so that the algorithms could be proven - make it safer than any proprietary solution developed in-door.
:68Domain-Driven-Design
%cartoon06.png
We have now discovered how {\i mORMot} offers you some technical bricks to play with, but it is up to you to build the house (castle?), according to your customer needs.
This is were @54@ - abbreviated DDD - patterns are worth looking at.
: Domain
What do we call {\i @**Domain@} here?\line {\i The domain represents a sphere of knowledge, influence or activity.}
As we already stated above, the domain has to be clearly identified, and your software is expected to solve a set of problems related to this domain.
DDD is some special case of {\i Model-Driven Design}. Its purpose is to create a {\i model} of a given domain. The code itself will express the model: as a consequence, any code refactoring means changing the model, and vice-versa.
\page
:91 Modeling
Even the brightest programmer will never be able to convert a real-life domain into its software code. What we can do is to create an {\i abstraction} system that describes selected aspects of a domain.
@**Model@ing is about {\i filtering} the reality, for a given use context: "All models are wrong, some are useful" {\i G. Box, statistician}.
:  Several Models to rule them all
As first consequence, {\i several models may coexist} for a given reality, depending of the knowledge level involved - what we call a {\i @**Bounded Context@}. Don't be afraid if the same reality is defined several times in your domain code: you should use only one {\f1\fs20 class} in a given context, but you may have another {\f1\fs20 class} defined in another context, with diverse attributes or methods.\line Just open {\i Google maps} for instance, and think how the same reality may be modeled depending on the zoom level, or you current view options. See also the M1, M2, M3 models as defined in {\i Meta-Object Facility}. When you define several models, you just need to clearly state the current model you are using.
Even models could be abstracted. This is what DDD does: the code itself is some kind of {\i meta-model}, conforming a given conceptual model to the grammar of a given programming language.
:  The state of the model
Most models express the reality in two dimensions:
- {\i Static}: to abstract a given {\i state} of the reality;
- {\i Dynamic}: to abstract how reality {\i evolves} (i.e. its behavior).
In both dimensions, we can clearly understand the purpose of {\i abstraction}.
Since it is impossible to model all the details of reality (e.g. describe a physical reality down to atomic / sub-atomic level), the static modeling will {\i forget} the non significant details, and focus on the essentials, for a given {\i knowledge level}, which is specific to a given context.
Similarly, most changes are continuous in the world, but dynamic modeling will create static snapshots of the reality (called {\i state transitions}), to embrace the deterministic nature of computers.
State always brings complexity to the model. As a consequence, our code should be as @*stateless@ as possible.\line Therefore:
- Try to always separate value and time in state;
- Reduce statefulness to the only necessary;
- Implement your logic as state machines instead of blocking code or sessions;
- Persistence should handle one-way transactions.
In DDD, {\i @*Value Objects@} and {\i @*Entity Objects@} are the mean to express a given system state. Immutable {\i Value Objects} define a static value. {\i Entity} refers to a given state of given identity (or reality).\line For instance, the same identity (named "John Doe") may be, at a given state, single and minor, then, at another state, married and adult. The model will help to express the given states, and the state transitions between them (e.g. John's marriage).
In DDD, the {\i @*Factory@} / {\i @*Repository@} / {\i @*Unit Of Work@} patterns will introduce transactional support in a stateless approach.
And in situations where a reality does change its state very often, with complex impacts on other components, DDD will model these state changes as {\i Events}. It could lead into introducing some @*Event-Driven@ Design even or @*Event Sourcing@ within the global model.
:170  Composition
In order to refine your model, you have two main tools at hand to express the model modularity:
- {\i Partitioning}: the more your elements have a {\i separated concern}, the better;
- {\i Grouping}: to express {\i constraints}, elements may be grouped - but usually, you should not put more than 6 or 8 elements in the same diagram, or your model may need to be refined.
In DDD, a lot of small objects have to be defined, in order to properly {\i partition} the logic. When we start with Object Oriented Programming, we are tempted to create huge classes with a lot of methods and parameters. This is a symptom of a weak model. We should always favor composition of small simple objects, just like the {\i Unix} tools philosophy or the {\i Single Responsibility Principle} - see @47@.
Some DDD experts also do not favor inheritance. In fact, inheriting may be also a symptom of some coupled context. Having two diverse realities sharing properties may be a bad design smell: if two or more classes inherit from one parent class, the state and behavior of the parent class may limit any future evolution of any of its children. In practice, trying to follow the {\i Open/Closed principle} - see @158@ - at {\f1\fs20 class} level may induce unexpected complexity, therefore reducing code maintainability.
In DDD, the {\i @*Aggregate Root@} is how you {\i group} your objects, in order to let constraints (e.g. business rules) to be modeled. {\i @*Aggregates@} are the main entry point to the domain, since they should contain, by design, the whole execution context of a given process. Their extent may vary during development, e.g. when a business rule evolves - remember that the same reality can appear several times in the same domain, but once per @*Bounded Context@. In other words, {\i Aggregates} could be seen as the smallest and biggest extent needed to express a given model context.
\page
: DDD model
It is now time to define which kind of {\i Model-Driven Design} is DDD:
%%ArchiDDDSyntax
:  Ubiquitous Language
{\i @**Ubiquitous Language@} is where DDD begins.
DDD expects the domain model to be expressed via a shared language, and used by all team members to connect their activities with the software. Those terms should be used in speech, writing, and any presentation or diagram.
In the real outside world, i.e. for the other 10th kind of people how do not know about binary, domain experts use company- or industry-standard terminology.
As developers, we have to understand this vocabulary and not only use it when speaking with domain experts but also see the same terminology reflected in our code. If the terms "class code" or "rate sets" or "exposure" are frequently used in conversation, we shall find corresponding class names in the code. In DDD, it is critical that developers use the business language in code consciously and as a disciplined rule. As a consequence, browsing the code should lead into a clear comprehension of the business model.
Domain experts will be the guard keepers of the consistency of this language, and its proper definition. Even if the terms are expected to be consistent, they are not to be written in stone, especially during the initial phase of software development. As soon as one domain activity cannot be expressed using the existing set of concepts, the model needs to be extended. Removing ambiguities and inconsistencies is a need, and will, very often, resolve several not-yet-identified software issues.
:  Value Objects and Entities
For the definition of your objects or internal data structures (what good programmers care about), you are encouraged to make a difference between several kind of objects. Following DDD, @*model@-level representation are, generally speaking, rich on behavior, therefore also of several families/species of objects.
Let us list the most high-level definitions of objects involved to define our DDD model:
- {\i @**Value Objects@} contain attributes (value, size) but no conceptual identity - e.g. money bills, or seats in a Rock concert, as they are interchangeable;
- {\i @**Entity objects@} are not defined by their attributes (values), but by their {\i thread of continuity}, signified by an identity - e.g. persons, or seats in most planes, as each one is unique and identified.
The main difference between {\i Value Objects} and {\i Entities} is that instances of the second type are tied to one reality, which evolves in the time, therefore creating a thread of continuity.
{\i Value objects are immutable} by definition, so should be handled as read-only. In other words, they are incapable of change once they are created.\line Why is it important that they be immutable? With {\i Value objects}, you're seeking side-effect-free functions, yet another concept borrowed by DDD to functional languages (and not available in most @*OOP@ languages, until latest concurrent object definition like in {\i Rust} or {\i Immutable Collections} introduced in C#/.NET 4.5). When you add $10 to $20, are you changing $20? No, you are creating a new money descriptor of $30. A similar behavior should be visible at code level.
{\i Entities} will very likely have an {\f1\fs20 ID} field, able to identify a given reality, and model the so-called {\i thread of continuity} of this identity. But this {\f1\fs20 ID} is an implementation detail, only used at {\i Persistence Layer} level: at the {\i Domain Layer} level, you should not access {\i Entities} individually, but via a special {\i Entity} bounded to a specific context, called {\i Aggregate Root} (see next paragraph).
When we define some objects, we should focus on making the implicit become {\i explicit}. For instance, if we have to store a phone number, we won't use a plain {\f1\fs20 string} type for it, but we will create a dedicated {\i Value object} type, making explicit all the behavior of its associated reality. Then we will be free to combine all types into explicit grouped types, on need.
:124  Aggregates
{\i @**Aggregates@} are a particular case of {\i Entities}, defined as collection of objects (nested {\i Values} and/or {\i Entities}) that are grouped together by a {\i root Entity}, otherwise known as an {\i @**Aggregate Root@}, which scope has been defined by a given execution context - see "{\i Composition}" above.
Typically, {\i Aggregates} are persisted in a database, and guarantee the consistency of changes by isolating its members from external objects (i.e. you can link to an aggregate via its ID, but you can not directly access to its internal objects). See @29@ which sounds like @http://martinfowler.com/bliki/AggregateOrientedDatabase.html
In practice, {\i Aggregates} may be the only kind of objects which will be persisted at the {\i Application layer}, before calling the domain methods: even if each nested {\i Entity} may have its own persistence method (e.g. one RDBMS table per Entity), {\i Aggregates} may be the unique access point to retrieve or update a given state. It will ensure so-called {\i @*Persistence Ignorance@}, meaning that domain should remain uncoupled to any low-level storage implementation detail.
DDD services may just permit remote access to {\i Aggregates} methods, where the domain logic will be defined and isolated.
:  Factory and Repository patterns
DDD then favors some patterns to use those objects efficiently.
The {\i @**Factory@ pattern} is used to create object instances. In strongly-typed OOP (like in {\i Delphi}, Java or C#), this pattern is in fact its {\i constructor} method and associated {\f1\fs20 class} type definition, which will define a fixed set of properties and methods at compilation time (this is not the case e.g. in JavaScript or weak-typed script languages, in which you can add methods and properties at runtime).\line In fact, Delphi is ahead of Java or C#, since it allows {\f1\fs20 virtual} constructors to be defined. Those {\f1\fs20 virtual} constructors are in fact a clean and efficient way of implementing a {\i Factory}, and also fulfill @*SOLID@ principles, especially the @159@: the parent class define an {\i abstract} constructor on which you rely, but the implementation will take place in the {\i overridden} constructor.
The {\i Factory pattern} can also be used to create {\f1\fs20 interface} instances - see @46@. Main benefit is that alternative implementations may be easily interchanged. Such abstraction helps testing - see @62@ - but also introduces interface-based services - see @63@.
{\i @**Repository@ pattern} is used to save and dispense each {\i Aggregate Root}.\line It matches the "@*Layer Supertype@" pattern (see above), e.g. via our {\i mORMot} {\f1\fs20 TSQLRecord} and {\f1\fs20 TSQLRest} classes and their Client-Server @*ORM@ features, or via dedicated repository classes - saving data is indeed a concern orthogonal to the model itself. DDD architects claim that persistence is infrastructure, not domain. You may benefit in defining your own repository interface, if the standard ORM / CRUD operations are not enough.
:156  DTO and Events to avoid domain leaking
The main DDD architecture principle - and benefit - is to isolate the domain code. As will be defined by the {\i @*Hexagonal architecture@} - see @155@, everything is made to ensure that the domain won't "leak" outside its core. The domain objects and services are the most precious part of any DDD project, especially in the long term, so proper isolation and uncoupling sound mandatory.
The {\i Aggregates} should always be isolated and stay at the {\i Application layer}, given access to its methods and nested objects via proper high-level remote {\i Services} - see @102@ - which should not be published directly to the outer world either.
In practice, if your domain is properly defined, most of your {\i Value Objects} {\i may} be sent to the outer world, without explicit translation. Even {\i Entities} may be transmitted directly, since their methods should not refer to nothing but their internal properties, so may be of some usefulness outside the domain itself.
But the real world may be rough and cruel, and optimism will better be replaced by some kind of pragmatism, and a pinch of cynicism. DDD experience told its pioneers (sometimes in a painful manner), that {\i @**Adapter@s} types should better be defined, especially at {\i Application layer} and {\i Presentation layer} levels.
As a result, a new family of objects will secure any DDD implementation:
- {\i Data Transfer Objects} (@**DTO@) are transmission objects, which purpose is to not send your domain across the wire (i.e. separate your layers, following the {\i @**Anti-Corruption Layer@} pattern). It encourages you to create gatekeepers (e.g. in the Application layer) that work to prevent non-domain concepts from leaking into your model.
- {\i @*Commands@} and {\i @*Events@} are some kind of DTO, since they communicate data about an event and they themselves encapsulate no behavior.
Using such dedicated types will eventually help {\i uncoupling} the domain, for several reasons:
- You can refactor your domain, without the need to modify the published interfaces, but just the tiny {\i Anti-Corruption layer}: no need for your customers to spend money upgrading their client applications, just because your domain changed; no fear to refine your precious domain code, in which you put all your money and expectations, just because it may be unpleasant to your customers.
- End-user application expectations won't pollute your domain. For instance, you will better define a per-customer set of public APIs, rather than exposing your domain services. In practice, a "one to rule them all" public API may sound like a good idea at first, but it will eventually end up as a monstrous, flat, unreadable and anemic interface, far away from @47@.
- Since the domain tends to be as generic as possible, its objects may sometimes be overkill to the end user applications: if some properties will never be used, or will always be void, why will you pollute your end user code, and waste bandwidth or resources? Just stick to what is needed.
- Dedicated types will help focusing on the needed use cases, so will ease documentation, maintainability, testing and integration with client applications: even translating your {\i Ubiquitous language} objects into more common or expected terms in the presentation layer will be beneficial.
- Consider that in your company, the {\i Domain} and {\i Infrastructure} layers may be maintained by your most valuable teams, whereas some less skilled developers (or even {\i offshore} teams) may be involved on {\i Application} and {\i Presentation} layers. Writing {\i adapter/translator} classes is not difficult, and will help your company focus and invest on where long term ROI is more likely to appear. Some access restrictions may therefore appear at source code level: it may be safe that only the wiser programmers will be allowed to modify the domain code, and even hide the domain implementation by publishing only its interfaces, protecting your most valuable intellectual property from being copied and stolen.
In {\i mORMot}, we try to let the framework do all the plumbing, letting those types be implemented via {\f1\fs20 interface}s over simple dedicated types like {\f1\fs20 record}s or {\i dynamic arrays} - see @154@ and @149@. So defining DTOs, {\i Commands} and {\i Events} in dedicated {\i Anti-Corruption layers} will be pretty much quick, easy and safe.
:102  Services
{\i @*Aggregate root@s} (and sometimes {\i Entities}), with all their methods, often end up as {\i state machines}, and the behavior matches accordingly.\line In the domain, since {\i Aggregate roots} are the only kind of entities to which your software may hold a reference, they tend to be the main access point of any process. It could be handy to publish their methods as stateless @*Service@s, isolated at {\i Application layer} level.
{\i Domain services pattern} is used to model primary operations.\line Domain Services give you a tool for modeling processes that do not have an identity or life-cycle in your domain, that is, that are not linked to one aggregate root, perhaps none, or several. In this terminology, services are not tied to a particular person, place, or thing in my application, but tend to embody processes. They tend to be named after verbs or business activities that domain experts introduce into the so-called {\i Ubiquitous Language}. If you follow the interface segregation principle - see @160@, your domain services should be exposed as dedicated client-oriented methods. Do not leak your domain! In DDD, you develop your {\i Application layer} services directly from the needs of your client applications, letting the {\i Domain layer} focus on the business logic.
{\i @**Unit Of Work@} can be used to maintain a list of objects affected by a business transaction and coordinates the writing out of changes and the resolution of concurrency problems.\line In short, it implements transactional process at Domain level, and may be implemented either at service or ORM level. It features so-called {\i @**Persistence Ignorance@}, meaning that your domain code may not be tied to a particular persistence implementation, but "hydrate" {\i Aggregate roots} class instances as abstractly as possible.\line A {\i @*dual-phase@ commit} approach - with some methods preparing and validation the data, then applying it by a dedicated {\f1\fs20 Commit} command in a second step - may be defined. In this pattern, the repository is just some simple storage, and data consistency will take place at domain level: for instance, you will not define any @*SQL@ constraints, but validate your data {\i before} storing the information. Your business rules should be written in high level domain code, and you may forget about the {\f1\fs20 FOREIGN KEY}, or {\f1\fs20 CHECK} SQL syntax flavors. As a result, you may safely change from a SQL database to a @*NoSQL@ engine, or even a {\f1\fs20 TObjectList}. You will be able to define and maintain any complex business rules, using the {\i Ubiquitous Language} of your domain. And a change of business logic will not impact the database metadata, which may be painful to modify.
The DDD Services may therefore be stateless for most of the time, but allowing some flavor of transactional process, when needed. The uppermost/peripheral architecture layers - i.e. {\i Application} or {\i Presentation Layers} - will ensure that those services will be propertly orchestrated. The application {\i workflows} will not be defined in the domain core itself, but in those outer layers, resulting in a cleaner, uncoupled architecture.
:155  Clean Uncoupled Architecture
If you follow properly the DDD patterns, your classic @7@ architecture will evolve into a so-called {\i @**Clean Architecture@} or {\i @**Hexagonal architecture@}.
Even if {\i physically}, this kind of architecture may still look like a classic {\i layered} design (with presentation on the top, business logic in the middle and a database at the bottom - and in this case we speak of {\i N-Layered Domain-Oriented Architecture}), DDD tries to isolate the {\i Domain Model} from any dependency, including technical details.
As a consequence, the {\i logical} architecture of any DDD solution should appear as such:
\graph mORMotDesignOnion Clean Uncoupled Domain-Oriented Architecture
subgraph cluster1 {
subgraph cluster2 {
subgraph cluster3 {
subgraph cluster4 {
\Unit Tests\Aggregates
\Unit Tests\Value¤Objects
\Unit Tests\Entities
\Entities\Value¤Objects
\Aggregates\Value¤Objects
\Aggregates\Entities
label="Domain\nModel";
}
"Third Party\nInterfaces";
\Repository¤Interfaces\Aggregates
\Domain¤Interfaces\Aggregates
\Domain¤Interfaces\Entities
\Domain¤Interfaces\Value¤Objects
label="Domain\nServices";
}
\Workflows\Third Party¤Interfaces\execute
\Workflows\Repository¤Interfaces\get/save¤objects
\Workflows\Domain¤Interfaces\use¤objects
label="Application\nServices";
}
\Behavior Tests\Workflows\validate
\Infrastructure\Workflows\implements
\User Interface\Workflows\exposes
\Database¤ORM/ODM\Infrastructure
\File¤System\Infrastructure
\Web¤Services\Infrastructure
\Mocks¤Stubs\Behavior Tests
\Fake¤Datasets\Behavior Tests
\Rich¤Client\User Interface
\Web¤AJAX\User Interface
label="Technical\nImplementations";
}
\
That kind of architecture is not designed in layers any more, but more like an Onion.
At the core of the bulb - sorry, of the system, you have the {\i Domain Model}.\line It implements all {\i @*Value Objects@} and {\i @*Entity Objects@}, including their state and behavior, and associated unit tests.
Around this core, you find {\i Domain @*Service@s} which add some more behavior to the inner model.\line Typically, you will find here abstract interfaces that provides persistence ({\i Aggregates} saving and retrieving via the {\i @*Repository@} pattern), let Domain objects properties and methods be defined (via the {\i @*Factory@} pattern), or access to third-party services (for service composition in a @*SOA@ world, or e.g. to send a notification email).
Then {\i Application @*Service@s} will define the {\i workflows} of all end-user applications.\line Even if the core Domain is to be as stable as possible, this outer layer is what will change more often, depending on the applications consuming the {\i Domain Services}. Typically, workflows will consist in deshydrating some {\i Aggregates} via the {\i Repository} interface, then call the {\i Domain} logic (via its objects methods, or for primary operations with wider Domain services), call any external service, and validate ("commit", following Unit-Of-Work or transactional terms) objects modifications. Some non data-centric process will also benefit from a {\i @*dual-phase@ commit} pattern, to allow safe orchestration of uncoupled domain and third party services.
Out on the edges you see User Interface, Infrastructure (including e.g. database persistence), and Tests. This outer layer is separated from the other three internal layers, which are sometimes called {\i Application Core}.\line This is where all technical particularities will be concentrated, e.g. where RDBMS / SQL / ORM mapping will be defined, or platform-specific code will reside. This is the right level to test your end-user workflows, e.g. using @**Behavior-Driven Development@ (abbreviated BDD), with the help of your Domain experts.
The premise of this Architecture is that {\i it controls coupling}. The main rule is that all coupling is toward the center: all code can depend on layers more central, but code cannot depend on layers further out from the core. This is clearly stated in the @%%mORMotDesignOnion@ diagram: just follow the arrows, and you will find out the coupling order. This architecture is unashamedly biased toward object-oriented programming, and it puts objects before all others.
This {\i Clean Architecture} relies heavily on the {\i Dependency Inversion} principle - see @47@. It emphasizes the use of {\f1\fs20 interface}s for behavior contracts, and it forces the externalization of infrastructure to dedicated implementation classes. The {\i Application Core} needs implementation of core interfaces, and if those implementing classes reside at the edges of the application, we need some mechanism for injecting that code at runtime so the application can do something useful. {\i mORMot}'s @63@ provide all needed process to access, even remotely, e.g. to persistence or any third party services, in an abstract way.
With {\i Clean Architecture}, the database is not the center of your logic, nor the bottom of your physical design - it is {\i external}. Externalizing the database can be quite a challenge for some people used to thinking about applications as "database applications", especially for {\i Delphi} programmers with a RAD / {\f1\fs20 TDataSet} background. With Clean Architecture, there are no database applications. There are applications that might use a database as a storage service but only though some external infrastructure code that implements an interface which makes sense to the application core. The domain could be even decoupled from any @*ORM@ pattern, if needed. Decoupling the application from the database, file system, third party services and all technical details lowers the cost of maintenance for the life of the application, and allows proper testing of the code, since all Domain {\f1\fs20 interface} types could be mocked on purpose - see @166@.
\page
:169 mORMot's DDD
:  Designer's commitments
Before going a bit deeper into the low-level stuff, here are some key sentences we should better often refer to:
- I shall collaborate with {\i domain experts};
- I shall focus on the {\i ubiquitous language};
- I shall not care about technical stuff or framework, but about modeling the {\i Domain};
- I shall make the implicit {\i explicit};
- I shall use end-user {\i scenarios} to get real and concrete;
- I shall not be afraid of defining {\i one model per context};
- I shall focus on my {\i Core Domain};
- I shall {\i let my Domain code uncoupled} to any external influence;
- I shall {\i separate values and time} in state;
- I shall {\i reduce statefulness} to the only necessary;
- I shall always {\i adapt my model} as soon as possible, once it appears inadequate.
As a consequence, you will find in {\i mORMot} no magic powder to build your DDD, but all the tools you need to focus on your business, without loosing time in re-inventing the wheel, or fixing technical details.
:  Defining objects in Delphi
How to implement all those DDD concepts in an object-oriented language like {\i Delphi}? Let's go back to the basics. Objects are defined by a {\i state}, a {\i behavior} and an {\i identity}. A {\i factory} helps creating objects with the same state and behavior.
In {\i Delphi} and most Object-Oriented (@**OOP@) languages - including C# or Java, each {\f1\fs20 @*class@} instance has the following behavior:
- {\i State} is defined by all its property / member values;
- {\i Behavior} are defined by all its methods;
- {\i Identity} is defined {\i by reference}, i.e. {\f1\fs20 a=b} is true only if {\f1\fs20 a} and {\f1\fs20 b} refers to the same object;
- {\i Factory} is in fact the {\f1\fs20 class} type definition itself, which will force each instance to have the same members and methods.
In {\i Delphi}, the {\f1\fs20 @*record@} type (and deprecated {\f1\fs20 object} type for older versions of the compiler) has an alternative behavior:
- {\i State} is also defined by all its property / member values;
- {\i Behavior} are also defined by all its methods;
- But {\i identity} is defined {\i by content}, i.e. {\f1\fs20 RecordEquals(a,b)} is true only if {\f1\fs20 a} and {\f1\fs20 b} have the same exact property values;
- {\i Factory} is in fact the {\f1\fs20 record} / {\f1\fs20 object} type definition itself, which will force each instance to have the same members and methods.
In practice, you may use either one of the two kinds of object types (i.e. either {\f1\fs20 class} or {\f1\fs20 record}), depending on the behavior expected by DDD patterns:
- DDD's {\i @*DTO@} may be defined as {\f1\fs20 record}, and directly serialized as JSON via text-based @51@ - as an alternative, you may consider using @80@;
- But other kinds of DDD objects , i.e. {\i @*Value Objects@}, {\i @*Entity Objects@} and {\i @*Aggregates@}, should better be defined as dedicated {\f1\fs20 class}, since {\f1\fs20 class} type definition offers more possibility than plain {\f1\fs20 record} structures. The framework defines some parent classes (e.e. {\f1\fs20 TSynPersistent} and {\f1\fs20 TSynAutoCreateFields}) which makes working with {\f1\fs20 class} instances almost as easy than stack-allocated {\f1\fs20 record} values.
:  Defining DDD objects in mORMot
When defining domain objects, we should always make the implicit {\i explicit}, i.e. writing one {\f1\fs20 class} type per reality in the model, in every {\i bounded context}. Thanks to {\i Delphi}'s strong typing, you will ensure that the Domain {\i Ubiquitous language} will appear in the code, and that your model will be expressed in a clean, uncoupled way.
If those {\f1\fs20 class} types are defined as plain PODO, even your domain experts - which may not know anything about writing code - may be part of the class definition: we usually write the domain objects and services with the domain experts, writing the code in real time during a meeting. The domain is therefore expressed as plain code, and experts are able to validate the workflows and properties of the model as soon as possible. Such coding sessions truly benefit of being a cooperative team work, not only coders'.
Once the domain model is stabilized, we may start implementing the interfaces using this common work as contract. In this implementation process, the {\i mORMot} framework offers a lot of tools to make it happen in a quick and efficient manner.
There are in fact two ways of implementing DDD objects as {\f1\fs20 class} types, in {\i mORMot}:
- Directly using the framework types, e.g. {\f1\fs20 TSQLRecord} specialized {\f1\fs20 class} for {\i Entities} or {\i Aggregates};
- Or relying of no framework structure, but clean @**PODO@s ({\i Plain Old Delphi Object} - see so-called POJO or POCO for Java or C#) {\f1\fs20 class} types, then use the {\f1\fs20 mORMotDDD.pas} unit for automatic marshalling.
Of course, the second option may be preferred, since it sounds like a better implementation path, uncoupled from the framework itself. Remember that DDD is mainly about {\i uncoupling} the Domain code from any external dependency, even from {\i mORMot} itself. You should better not be forced to use the framework ORM, if you have some existing legacy SQL statements, for instance.
:   Use framework types for DDD objects
If you want to directly use framework structure, DDD's {\i @*Value Objects@} are probably meant to be defined as {\f1\fs20 record}, with methods (i.e. in this case as {\f1\fs20 object} for older versions of {\i Delphi}). You may also use {\f1\fs20 TComponent} or {\f1\fs20 TSQLRecord} classes, ensuring the {\f1\fs20 published} properties do not have setters but just {\f1\fs20 read F...} definition, to make them read-only, and, at the same time, directly serializable.\line If you use {\f1\fs20 record} / {\f1\fs20 object} types, you may need to customize the JSON serialization - see @51@ - when targeting AJAX clients, especially for any version prior to {\i Delphi} 2010 (by default, {\f1\fs20 record}s are serialized as binary + @*Base64@ encoding due to the lack of @*enhanced RTTI@, but you can define easily the record serialization e.g. from text). Note that since {\f1\fs20 record} / {\f1\fs20 object} defines in {\i Delphi} {\i by-value} types (whereas {\f1\fs20 class} defines {\i by-reference} types - see previous paragraph), they are probably the cleanest way of defining {\i Value Objects}.
In this context, DDD's {\i @*Entity objects@} could inherit from {\f1\fs20 TSQLRecord}. It will give access to a whole set of methods supplied by {\i mORMot}, implementing some kind of "@**Layer Supertype@", as explained by Martin Fowler.
Finally, DDD's {\i @*Aggregates@} will benefit of using {\i mORMot}'s @3@. {\i Entities} will be stored as regular {\f1\fs20 TSQLRecord}, e.g. using @70@ cardinality, as available from the framework.
For most simple cases, this solution may be just good enough. But it may have the drawback of coupling your {\i Domain logic} with {\i mORMot} internals. Your Domain will eventually be polluted by the framework implementation details, which should better be avoided.
:   Define uncoupled DDD objects
In order to uncouple our {\i Domain} code from its persistence layer, {\i mORMot} offers some dedicated types and units to use @*PODO@ {\f1\fs20 class} definitions within your DDD core.
You may use regular {\f1\fs20 TPersistent} as parent class, but you may consider using {\f1\fs20 TSynPersistent} and {\f1\fs20 TSynAutoCreateFields} fields instead - we will see soon their benefit.
Let's start from existing code, available in the {\f1\fs20 SQLite3\\DDD\\dom} sub-folder of the framework source code repository, in the {\f1\fs20 dddDomUserTypes.pas} unit. This unit defined some reusable class types, able to store user information, in a clean DDD way.
:   Specialize your simple types
Each reality in this unit will have its own type definition, using the extended pascal syntax, even for simple types like {\f1\fs20 string} or {\f1\fs20 integer}:
!type
!  TSpecifiedType = type TParentType;
You may not be familiar with this syntax. But it is a pretty powerful mean of defining your DDD model with a plain pascal syntax. Here {\f1\fs20 TSpecifiedType} is defined as a specific type, which will behave like {\f1\fs20 TParentType}, but {\i strong-typing} will apply in your code, so that the compiler will complain if you pass e.g. a {\f1\fs20 TParentType} instead of a {\f1\fs20 TSpecifiedType} as a {\f1\fs20 var} parameter. It will help to resolve some ambiguities when transmitting information.
For instance, in the {\f1\fs20 dddDomUserTypes.pas} unit, you may see:
!type
!  TLastName = type RawUTF8;
!  TFirstName = type RawUTF8;
!  TMiddleName = type RawUTF8;
Thanks to those type definitions, you will be able to make a difference between a last name, a first name and a middle name. We used {\f1\fs20 RawUTF8} as parent type, but we may have used {\f1\fs20 string}. Since we wanted our code to work seamlessly with all versions of Delphi and FPC, we rather rely on {\f1\fs20 RawUTF8} - see @32@.
Once compiled, there won't be any difference between the three types, which will behave like a {\f1\fs20 RawUTF8}. But at compile time, and in your Domain source code, you will be able to know exactly which reality is stored in a given variable.
So instead of this method definition:
!  function UserExists(const aUserName: RawUTF8): boolean;
You will rather write:
!  function UserExists(const aUserName: TLastName): boolean;
With such a method signature, we will ensure that we won't supply a {\f1\fs20 TFirstName} or a {\f1\fs20 TPetName} by mistake.
It may sound like a small enhancement, but be sure that it will increase your code safety, and expressiveness. One of the biggest failure in NASA history was {\i Mars Climate Orbiter}. A variable type error burn up a $327.6 million project in minutes, when one engineering group working on the thrusters measured in English units of pounds-force seconds, whereas the others used metric Newton-seconds. The result of that inattention is now lost in space, possibly in pieces.
Remember when our physic teachers leaped all over answers that consisted of a number. If the answer was 2.5, they will take their red pens and write "2.5 what? Weeks? Puppies? Demerits?" And proceed to mark the answer wrong. In our DDD code, we should rather follow this rule, and try to make the implicit {\i explicit}.
:   Define your PODO classes
Main point is first to define your DDD Objects as plain Delphi {\f1\fs20 class} types - the famous @*PODO@s, following the {\i Ubiquitous Language}. We will in fact define {\i @*Value Objects@} {\f1\fs20 class} types, which may be grouped and nested to become {\i @*Entity Objects@} or {\i @*Aggregates@}.
To define a {\f1\fs20 TPerson} object, able to modelize a person identity, we may write the following classes:
!type
!  /// Person full name
!  TPersonFullName = class(TSynPersistent)
!  protected
!    fFirst: TFirstName;
!    fMiddle: TMiddleName;
!    fLast: TLastName;
!  public
!    function Equals(another: TPersonFullName): boolean; reintroduce;
!    function FullName(country: TCountryIdentifier=ccUndefined): TFullName; virtual;
!  published
!    property First: TFirstName read fFirst write fFirst;
!    property Middle: TMiddleName read fMiddle write fMiddle;
!    property Last: TLastName read fLast write fLast;
!  end;
!
!  /// Person birth date
!  TPersonBirthDate = class(TSynPersistent)
!  protected
!    fDate: TDateTime;
!  public
!    function Equals(another: TPersonBirthDate): boolean; reintroduce;
!    function Age: integer; overload;
!    function Age(FromDate: TDateTime): integer; overload;
!  published
!    property Date: TDateTime read fDate write fDate;
!  end;
!
!  /// Person object
!  TPerson = class(TSynAutoCreateFields)
!  protected
!    fBirthDate: TPersonBirthDate;
!    fName: TPersonFullName;
!  public
!    function Equals(another: TPerson): boolean; reintroduce;
!  published
!    property Name: TPersonFullName read fName;
!    property Birth: TPersonBirthDate read fBirthDate;
!  end;
First of all, you will see that we inherit from {\f1\fs20 TSynPersistent} and {\f1\fs20 TSynAutoCreateFields}. The benefit of those classes are the following:
- {\f1\fs20 TSynPersistent} has a {\f1\fs20 virtual} constructor, and a little less overhead than {\f1\fs20 TPersistent}, so may be preferred, especially when we will use @161@;
- {\f1\fs20 TSynAutoCreateFields} inherits from {\f1\fs20 TSynPersistent}, and its overridden {\f1\fs20 Create} will allocate all {\f1\fs20 published} class properties auto-magically - whereas its overridden {\f1\fs20 Destroy} will release those instances for you. As such, inheriting from {\f1\fs20 TSynAutoCreateFields} makes it a perfect fit for a {\i Value Object}, nesting sub objects as properties;
- Both have the RTTI enabled, so all published properties will be easily serialized as JSON (when used as @*DTO@), or persisted later on on a database, when joined as {\i Aggregate Roots}.
In the above code, we defined {\f1\fs20 TPerson.Name} as a {\f1\fs20 TPersonFullName} class. So that we may use {\f1\fs20 aPerson.Name.First} or {\f1\fs20 aPerson.Name.Last} or even the runtime-computed {\f1\fs20 aPerson.Name.FullName} method which is able to display the full name, depending on per-country culture. We also reintroduced the {\f1\fs20 Equals()} method, which will allow to compare the objects per value, and not per reference.
Even if the birth date is just a date, we introduced a dedicated {\f1\fs20 TPersonBirthDate} class. The benefit is to have the overloaded {\f1\fs20 Age()} methods, which are pretty convenient in practice.
Once serialized as JSON, a {\f1\fs20 TPerson} content may be:
${
$ "Name": {
$  "First": "John",
$  "Middle": "",
$  "Last": "Smith"
$ },
$ "Birth": {
$  "Date": "1972-10-29"
$ }
$}
During the modelization phase, you will just define such {\f1\fs20 class} types, trying to reflect DDD's {\i Ubiquitous Language} into regular Delphi classes.
Take a look at the {\f1\fs20 dddDomUserTypes.pas} unit, to identify such patterns, and how we may be able to define an application user, gathering our {\f1\fs20 TPerson} class with a {\f1\fs20 TAddress}, in which a {\f1\fs20 TCountry} class will be used to store the corresponding country:
!  /// a Person object, with some contact information
!  // - an User is a person, in the context of an application
!  TPersonContactable = class(TPerson)
!  protected
!    fAddress: TAddress;
!    fPhone1: TPhoneNumber;
!    fPhone2: TPhoneNumber;
!    fEmail: TEmailAddress;
!  public
!    function Equals(another: TPersonContactable): boolean; reintroduce;
!  published
!    property Address: TAddress read fAddress;
!    property Phone1: TPhoneNumber read fPhone1 write fPhone1;
!    property Phone2: TPhoneNumber read fPhone2 write fPhone2;
!    property Email: TEmailAddress read fEmail write fEmail;
!  end;
You can see that we did not pollute the {\f1\fs20 class} definition with any detail about persistence. What we did by now was to define a plain {\i Value Object}. We did not even specify that this {\f1\fs20 class} may be any {\i Entity}, nor introduce a primary key to identify it from a single access point. We found this way much cleaner that the approach of most other Java or C# DDD frameworks, which usually require to inherit from a parent {\f1\fs20 Entity} class, or use {\i attributes} to define the persistence expectations (like the primary key). We think that the domain types should not be polluted with those implementation details, and focus on expressing the model.
We will finally define a {\f1\fs20 TUser} {\i Entity} (or {\i Aggregate Root}), inheriting from {\f1\fs20 TPersonContactable}, i.e. modelizing any application user account with all its personal information, with a flag to testify that its email was validated:
!  TUser = class(TPersonContactable)
!  private
!    fLogonName: TLogonName;
!    fEmailValidated: TDomUserEmailValidation;
!  published
!    property LogonName: TLogonName read fLogonName write fLogonName;
!    property EmailValidated: TDomUserEmailValidation read fEmailValidated write fEmailValidated;
!  end;
Such a {\f1\fs20 TPersistent}-inheriting class could be used as a {\i Value Object} (or even a @*DTO@), but become an {\i Entity} or {\i Aggregate} in the bounded context of the user account personal information. In order to store this data, we will now define an {\f1\fs20 interface}, implementing a {\i Persistence Service}.
:167   Store your Entities in CQRS Repositories
When persisting our precious DDD Objects, the framework tries to follow some DDD patterns:
- Define {\i Aggregate Root} (or {\i Entities}) from {\i Value Objects}, as practical data context for storing the information;
- Use a {\i @**Repository@} service to store those {\i @**Aggregates@} instances;
- Follow {\i @**CQRS@} ({\i Command Query Responsibility Segregation}) via a dedicated dual {\f1\fs20 interface}, splitting reads ({\i Queries}) and writes ({\i Commands}) in the {\i Repository} contract;
- Use {\i @*Factory@} to instantiate {\i CQRS Repository} contracts on need.
In practice, we will use a {\i Factory} to create {\i Repository} {\f1\fs20 class} instances implementing the {\i CQRS} service methods, defined as a hierachy of {\f1\fs20 interface} types, for a given {\i Aggregate Root}.\line Let's start from an example, i.e. implement {\i CQRS Repository} services for our {\f1\fs20 TUser} {\f1\fs20 class}.
:    CQRS Interfaces
The {\f1\fs20 mORMotDDD.pas} unit defines the following {\f1\fs20 interface}, which will benefit of being the root {\f1\fs20 interface} of all {\i Repository} services:
!type
!  ICQRSService = interface(IInvokable)
!    ['{923614C8-A639-45AD-A3A3-4548337923C9}']
!    function GetLastError: TCQRSResult;
!    function GetLastErrorInfo: variant;
!  end;
This interface does nothing but allowing a generic access to the last error which occurred. This will be used instead of {\f1\fs20 Exception}, via the {\f1\fs20 TCQRSResult} enumeration, as a safe way of handling errors in a remote {\i Service}.
Exceptions are very convenient when running code in a process, but are difficult to handle over a remote connection, since the execution context is spread on both client and server sides. It is very difficult to propagate an {\f1\fs20 exception} raised on the server side to the client side, without leaking the server implementation. For instance, the @*SOAP@ standard provides a way of transmitting execution errors as dedicated XML messages - but it turns out to be a very verbose and complex path.
In {\i mORMot}, we defined a generic way of sending errors to the client side, for CQRS Services. By convention, any method will be defined as a {\f1\fs20 function}, returning its execution state as a {\f1\fs20 TCQRSResult} enumeration. If {\f1\fs20 cqrsSuccess} is returned, no error did happen on the server side, and execution may continue on the client side. Otherwise, an error "kind" is specified in the {\f1\fs20 TCQRSResult} transmitted value, and additional information is available as {\f1\fs20 string} or a @80@ in the {\f1\fs20 ICQRSService.GetLastErrorInfo} method. This allows to safely handle any kind of execution error on the client side, without the need to define dedicated exceptions. As we already stated about @165@, {\f1\fs20 exception} should be {\i exceptional} - please refer to this paragraph for more details, including the benefit of that any stubbed or mocked interface will return {\f1\fs20 cqrsSuccess} (i.e. 0) by default, so let the test pass.
For our {\f1\fs20 TUser} {\i CQRS Repository} service, we will therefore define two {\f1\fs20 interface} types, one inheriting from {\f1\fs20 ICQRSService} for the {\i Queries} methods, and another one inheriting from this later interface to define the {\i Commands} methods:
\graph DDDCQRSInterface CQRS Repository Service Interface for TUser
\IDomUserCommand\IDomUserQuery
\IDomUserQuery\ICQRSService
\Write Operations\Read Operations
\Read Operations\Error Handling
\
In {\f1\fs20 dddDomUserCQRS.pas}, we therefore defined two {\f1\fs20 interface} types, one {\f1\fs20 IDomUserQuery} for the read operations (i.e. {\i Queries}) of {\f1\fs20 TUser} aggregates, and an inherited {\f1\fs20 IDomUserCommand} for the write operations (i.e. {\i Commands}) of {\f1\fs20 TUser} aggregates.
We may argue that {\f1\fs20 IDomUserCommand} inheriting from {\f1\fs20 IDomUserQuery} is actually a violation of the {\i Command Query Responsibility Segregation} principle. Here, {\i Commands} are tied to {\i Queries}. Of course, we may have defined two diverse interfaces, both inheriting from {\f1\fs20 ICQRSService} as parent:
\graph DDDCQRSInterface2 CQRS Dogmatic Repository Service Interface for TUser
\IDomUserCommand\ICQRSService
\IDomUserQuery\ICQRSService
\Write Operations\Error Handling
\Read Operations\Error Handling
\
Nothing prevent you from doing this. But in our case, especially with the {\i mORMot} underlying ORM, or a RDBMS database, the benefit is not obvious - sounds more like a dogmatic approach. To {\i update} a resource, you will need two interfaces: one {\f1\fs20 IDomUserQuery} instance to retrieve the existing value object, then one {\f1\fs20 IDomUserCommand} to modify it. From our pragmatic point of view, it is not mandatory. Also note that {\f1\fs20 interface} inheritance may differ from actual implementation {\f1\fs20 class} inheritance. {\f1\fs20 IDomUserCommand} may inherit from {\f1\fs20 IDomUserQuery}, but, e.g. if performance matters, you may still be able to implement a plain {\f1\fs20 IDomUserQuery} service with a dedicated {\f1\fs20 class}, on a separated database. In our case, {\f1\fs20 interface} inheritance is a common way of increasing code reuse. So if you want to be dogmatic about CQRS, you could - but only if it is worth the effort.
:    Queries Interface
Since we will separate queries and commands, we will first define the interface for actually reading {\f1\fs20 TUser} information:
!type
!  IDomUserQuery = interface(ICQRSService)
!    ['{198C01D6-5189-4B74-AAF4-C322237D7D53}']
!    function SelectByLogonName(const aLogonName: RawUTF8): TCQRSResult;
!    function SelectByEmailValidation(aValidationState: TDomUserEmailValidation): TCQRSResult;
!    function SelectByLastName(const aName: TLastName; aStartWith: boolean): TCQRSResult;
!    function Get(out aAggregate: TUser): TCQRSResult;
!    function GetAll(out aAggregates: TUserObjArray): TCQRSResult;
!    function GetNext(out aAggregate: TUser): TCQRSResult;
!    function GetCount: integer;
!    function HowManyValidatedEmail: integer;
!  end;
As we stated previously, all those methods do return a {\f1\fs20 TCQRSResult} enumeration, which will be used on the service consumer side to notify on any execution error.
Instances of those interface will in fact have a limited life-time. To access the {\f1\fs20 TUser} persistence layer, a CQRS {\f1\fs20 interface} will be injected - via @161@, then allow to handle one or several {\f1\fs20 TUser} instances.
For queries, you could use {\f1\fs20 IDomUserQuery.SelectByLogonName}, {\f1\fs20 IDomUserQuery.SelectByLastName} or {\f1\fs20 IDomUserQuery.SelectByEmailValidation} methods to initialize a request. As you can see, there is no mention of primary key or {\f1\fs20 ID} in this {\f1\fs20 interface} definition. Even if under the hood, the implementation {\i may} use our ORM, and a {\f1\fs20 TSQLRecord} with its {\f1\fs20 TSQLRecord.ID: TID} property, the CQRS interface themselves make not those implementation details appear - unless it will be necessary. In our use case of an application targeting a single user, it is enough to be able to retrieve a user by its logon name, or by its last name.
If the {\f1\fs20 Select*} method executed without error (i.e. returned {\f1\fs20 cqrsSuccess}), we can later on retrieve the content by calling:
- {\f1\fs20 IDomUserQuery.Get} for filling the properties of a single already existing {\f1\fs20 TUser} object);
- {\f1\fs20 IDomUserQuery.GetAll} to return a list of {\f1\fs20 TUser} instances - for storage, we will use a {\f1\fs20 TUserObjArray} dynamic array, which should be released by the caller using {\f1\fs20 ObjArrayClear()} on the result variable;
- {\f1\fs20 IDomUserQuery.GetNext} to retrieve the actual matching {\f1\fs20 TUser}, one by one, following the principle of a database {\i cursor};
- {\f1\fs20 IDomUserQuery.GetCount} will return the number of items matching the {\f1\fs20 Select*}.
Since the {\f1\fs20 IDomUserQuery interface} has a lifetime, you could call {\f1\fs20 IDomUserQuery.Get} or {\f1\fs20 IDomUserQuery.GetAll} several times after a single {\f1\fs20 Select*}. Note that in the common ORM-based implementation we will define below, the {\f1\fs20 TUser} information is actually {\i retrieved} and stored in memory by the {\f1\fs20 Select*} method.
Note that in the {\f1\fs20 IDomUserQuery} contract, the {\f1\fs20 IDomUserQuery.HowManyValidatedEmail} method, on the other hand, is stateless, and could be used without any prior {\f1\fs20 Select*}. Such methods may appear, depending on the Domain expectations.
The main point here is that, when defining your CQRS interface, you should focus on {\i which} data you need to access, in the most convenient way for you, and forget about the real persistence implementation - i.e. {\i how} data is stored. This is called, in DDD methods, as {\i @**Persistence Ignorance@}, and is a very convenient way of uncoupling your business logic from actual technical details. If you was never asked by your commercials to support a new database engine, or even be able to switch from a SQL to a NoSQL storage, or an existing legacy proprietary obscure database used by a given customer... you are a lucky programmer, but - you know - it happens in real life!
Another advantage of starting from what you need in your domain, by using {\f1\fs20 interface} types as contracts, is that you will probably focus on the domain, and may avoid the risk of an {\i @**anemic domain model@} symptom, which appears when your persistence service is just a CRUD operation in disguise. If we need only CRUD operations, an ORM, or even plain SQL is enough. But if we want to have our domain code follow the ubiquitous language, and stick to the use cases of our business model, we should better design the persistence this way.
Last but not least, you will be able to {\i mock} or {\i stub} the persistence service - see @166@, so ease unit test of your {\i Domain} code, without any dependency to any actual database layer. Following {\i Test Driven Design}, you will even be able to write the Domain core tests first, validate all your interfaces, even write the {\i Application layer} and test it with the current mock-up of the end-user application, and eventually finalize and tune the SQL or NoSQL storage at the final step, when the whole workflow is stabilized. It will help testing sooner, therefore fix sooner, and... hopefully release sooner.
:    Commands Interface
Following the CQRS ({\i Command Query Responsibility Segregation}) principle, we defined the write operations (i.e. {\i Commands}) in a separate {\f1\fs20 interface}. This type will inherit from {\f1\fs20 IDomUserQuery}, since it may be convenient to be able to first {\i read} the {\f1\fs20 TUser}, for instance before applying a modification to the stored information, like updating existing data, or adding some a missing entry.
!type
!  IDomUserCommand = interface(IDomUserQuery)
!    ['{D345854F-7337-4006-B324-5D635FBED312}']
!    function Add(const aAggregate: TUser): TCQRSResult;
!    function Update(const aUpdatedAggregate: TUser): TCQRSResult;
!    function Delete: TCQRSResult;
!    function DeleteAll: TCQRSResult;
!    function Commit: TCQRSResult;
!    function Rollback: TCQRSResult;
!  end;
The main method of this {\i Command} interface is {\f1\fs20 Commit}. Following the {\i @**dual-phase@ commit} pattern, nothing will be written to the actual persistence storage unless this {\f1\fs20 IDomUserCommand.Commit} method is actually called.
In short, you query then update your data using the other {\f1\fs20 Add}/{\f1\fs20 Update}/{\f1\fs20 Delete}/... methods, then you run {\f1\fs20 Commit}.
For instance, to modification an existing record, you will call:
- {\f1\fs20 IDomUserQuery.SelectByLogonName};
- {\f1\fs20 IDomUserCommand.Update};
- {\f1\fs20 IDomUserCommand.Commit}.
If the logon name is unknown, an error will raise at the first step. If the updated modification transmitted at the second step is invalid (i.e. you forgot to fill a mandatory field, or a value which should be unique, like a serial number, appear to exist already), then another error will be reported. But even after a successful {\f1\fs20 Update}, nothing will be stored in the database. Why? Because in most use cases, you will probably need to synchronize several operations: for instance, you may have to send an email, or call a third-party service, and write the new data only if everything was right. As such, you will need a two-phase write operation: first, you prepare and validate your data on each involved service, then, once everyone did give its green light, you eventually launch the process, which is, in the case of a persistence layer, calling {\f1\fs20 Commit}. In a real application, an unexpected low-level error may happen during the {\i Commit} phase - e.g. a network failure, a concurrency issue, or a problem between a chair and a keyboard - but it will not be likely to happen often. The {\i dual-phase} commit will ensure that most errors will be identified during the first phase, using our ORM's @56@ abilities.
Of course, if you want to run the {\f1\fs20 IDomUserCommand.Add} method, no prior {\f1\fs20 IDomUserQuery.Select*} call is mandatory. But for {\f1\fs20 Update} and {\f1\fs20 Delete} or {\f1\fs20 DeleteAll} commands, you will need first to define the data extend you will work on, by a previous call to {\f1\fs20 Select*}.
To use those CQRS interfaces, you could use @*IoC@ as usual:
!var cmd: IDomUserCommand;
!    user: TUser;
!    itext: RawUTF8;
!...
!!  aServer.Services.Resolve(IDomUserCommand,cmd);
!  user := TUser.Create;
!  try
!    for i := 1 to MAX do begin
!      UInt32ToUtf8(i,itext);
!      user.LogonName := '  '+itext;
!      user.EmailValidated := evValidated;
!      user.Name.Last := 'Last'+itext;
!      user.Name.First := 'First'+itext;
!      user.Address.Street1 := 'Street '+itext;
!      user.Address.Country.Alpha2 := 'fr';
!      user.Phone1 := itext;
!!      if cmd.Add(user)<>cqrsSuccess then
!          raise EMyApplicationException.CreateFmt('Invalid data: %s',[cmd.GetLastErrorInfo]);
!    end;
!    // here nothing is actually written to the database
!!    if cmd.Commit<>cqrsSuccess then
!        raise EMyApplicationException.CreateFmt('Commit error: %s',[cmd.GetLastErrorInfo]);
!    // here everything has been written to the database
!  finally
!    user.Free;
!  end;
This {\i dual-phase} commit appears to be a clean way of implement the @100@. Under the hood, when used with our ORM - as we will now explain - {\i @*Unit Of Work@} will be expressed as a {\f1\fs20 I*Command} service, uncoupled from the persistence layer it runs on.
:    Automated Repository using the ORM
As you may have noticed, we did just defined the {\f1\fs20 interface} types we needed. That is, we have the {\i contract} of our persistence services, but no actual implementation of it. As such, those {\f1\fs20 interface} definitions are useless. Luckily for us, the {\f1\fs20 mORMotDDD.pas} unit offers an easy way to implement those using @3@, with minimal coding.
:     DDD / ORM mapping
First we will need to map our domain object (i.e. our {\f1\fs20 TUser} instance and its properties) into a {\f1\fs20 TSQLRecord}. We may do it by hand, but you may find an handy way. Just run the following in the context of your application:
! TDDDRepositoryRestFactory.ComputeSQLRecord(TUser);
This {\f1\fs20 class procedure} will create a {\f1\fs20 ddsqlrecord.inc} file in the executable folder, containing the needed field definition, with one {\f1\fs20 TSQLRecord} type corresponding to each hierarchy level of the original {\f1\fs20 TPersistent} definition. Nested fields will be defined as a single column in the {\f1\fs20 TSQLRecord}, e.g. {\f1\fs20 Address.Country.Iso} will be flattened as a {\f1\fs20 Address_Country} property.
So if we follow the class hierarchy, we will have:
\graph DDDCQRSORMMapping CQRS Class Hierarchy Mapping for ORM and DDD Entities
\TPerson\TPersonContactable
\TPersonContactable\TUser
\TSQLRecordPerson\TSQLRecordPersonContactable
\TSQLRecordPersonContactable\TSQLRecordUser
\
Which will be defined as such in the {\f1\fs20 ddsqlrecord.inc} generated content:
!type
!  TSQLRecordPerson = class(TSQLRecord)
!  ...
!  published
!    property Name_First: RawUTF8 read fFirst write fFirst;
!    property Name_Middle: RawUTF8 read fMiddle write fMiddle;
!    property Name_Last: RawUTF8 read fLast write fLast;
!    property Birth: TDateTime read fBirthDate;
!  end;
!
!  TSQLRecordPersonContactable = class(TSQLRecordPerson)
!  ...
!  published
!    property Address_Street1: RawUTF8 read fStreet1 write fStreet1;
!    property Address_Street2: RawUTF8 read fStreet2 write fStreet2;
!    property Address_CityArea: RawUTF8 read fCityArea write fCityArea;
!    property Address_City: RawUTF8 read fCity write fCity;
!    property Address_Region: RawUTF8 read fRegion write fRegion;
!    property Address_Code: RawUTF8 read fCode write fCode;
!    property Address_Country: integer read fCountry;
!    property Phone1: RawUTF8 read fPhone1 write fPhone1;
!    property Phone2: RawUTF8 read fPhone2 write fPhone2;
!    property Email: RawUTF8 read fEmail write fEmail;
!  end;
!
!  TSQLRecordUser = class(TSQLRecordPersonContactable)
!  ...
!  published
!    property LogonName: RawUTF8 read fLogonName write fLogonName;
!    property EmailValidated: TDomUserEmailValidation read fEmailValidated write fEmailValidated;
!  end;
In practice, the following property will need to be tuned as such:
!    property LogonName: RawUTF8 read fLogonName write fLogonName
!!      stored AS_UNIQUE;
Take a look at the {\f1\fs20 dddInfraRepoUser.pas} and {\f1\fs20 dddDomUserTypes.pas} units to make a comparison between the DDD objects and their corresponding {\f1\fs20 TSQLRecord*} types.
You may wonder why we will introduce a separate level of classes, between the DDD Aggregates and the database engine. Why not directly persist the Domain objects (as most DDD implementations do)?
In fact, our approach has several benefits:
- Most of the time, simple mapping will be done automatically: once you called {\f1\fs20 TDDDRepositoryRestFactory.ComputeSQLRecord}, there is a very little additional coding to be done;
- But you still have access to the full mapping process, not using attributes (which may sound convenient, but are {\i polluting} the DDD classes definition), but at {\i Persistence Service} method level;
- You could persist the same DDD classes as {\i Value Objects}, {\i Entities} or {\i Aggregates}, depending on the use context, by a custom mapping over a dedicated {\i Persistence Service} - your domain objects are uncoupled from their use context - remember that the same Value Object may become an Aggregate, or an Entity, depending on the context: why define again and again the same classes? just reuse the same tuned types via @170@;
- No need to inherit your DDD classes from a parent {\i Entity} class, or pollute it with an {\f1\fs20 ID} field (as most DDD implementations do);
- {\f1\fs20 TSQLRecord} allows to be truly persistent agnostic: you may do the storage on a regular RDBMS engine, on a @*NoSQL@ database, or in memory, at runtime, without touching your DDD objects;
- Practice did show that introducing ORM concepts at DDD classes level: just think about how the ID field may break your modelization, since the same object may be a {\i Value Object} in a context (so without any ID), but an {\i Entity} or an {\i Aggregate} in another context (so an ID is probably needed there) - it does indeed break the {\i @*Persistence Ignorance@} pattern, and tend to produce an {\i @*anemic domain model@}, i.e. CRUD operations in disguise;
- {\f1\fs20 TSQLRecord} classes give you direct access to how your data will be actually stored: most ORMs, when dealing with complex classes (like our Domain objects), tend to hide the mapping complexity, and therefore make it difficult to debug and tune the storage itself: which object field is mapped to which column? which tables are involved and joined for the queries? - whereas {\f1\fs20 TSQLRecord} make it clear how data will actually been stored: you may consider the {\f1\fs20 TSQLRecord} properties as a map of the SQL storage columns, or as the document stored in a NoSQL engine - database reuse and tuning will definitively be easier, when the {\f1\fs20 TSQLRecord} type definition shows you e.g. where the indexes should be created;
- You are not tied to use {\f1\fs20 TSQLRecord}: you can easily define a {\f1\fs20 mORMotDDD.pas} CQRS repository service fully abstracted from {\i mORMot}'s ORM, e.g. using existing tuned SQL statements, or any other mean of storage;
- Also consider that you are able to easily @166@ the CQRS persistence service, whereas a direct ORM-oriented implementation will force you to create fake databases.
If you worry about performance of adding such a layer, you may be confident it won't be a bottleneck: the CQRS mapping shares the same code than the framework ORM for RTTI and marshalling. Mapping process is just a fast loop over the properties, using cached RTTI, and assigning all content by reference, avoiding most memory allocations or content transformation.
:     Define the Factory
Since the generated {\f1\fs20 TSQLRecordUser} type follows known conventions, the {\f1\fs20 mORMotDDD.pas} unit is able to do almost all the persistence work in an automated way, by inheriting of two classes:
- Defining a {\i Repository Factory} (i.e. a class able to generate {\f1\fs20 IDomUserQuery} or {\f1\fs20 IDomUserCommand} instances on requests) by inheriting from {\f1\fs20 TDDDRepositoryRestFactory}
- Defining the actual {\f1\fs20 IDomUserCommand} methods by inheriting from {\f1\fs20 TDDDRepositoryRestCommand}, and using high level protected methods to access the {\f1\fs20 TUser} from internal {\f1\fs20 TSQLRecordUser} ORM values.
First of all, we define the Factory:
!type
!  TInfraRepoUserFactory = class(TDDDRepositoryRestFactory)
!  public
!    constructor Create(aRest: TSQLRest; aOwner: TDDDRepositoryRestManager=nil); reintroduce;
!  end;
!
!constructor TInfraRepoUserFactory.Create(aRest: TSQLRest;
!  aOwner: TDDDRepositoryRestManager);
!begin
!  inherited Create(IDomUserCommand,TInfraRepoUser,TUser,aRest,TSQLRecordUser,aOwner);
!  AddFilterOrValidate(['*'],TSynFilterTrim.Create);
!  AddFilterOrValidate(['LogonName'],TSynValidateNonVoidText.Create);
!end;
As you can see, the main point of this {\f1\fs20 constructor} is to supply the right parameters to the inherited {\f1\fs20 TDDDRepositoryRestFactory.Create}:
- We would like to implement a {\f1\fs20 IDomUserCommand} contract - and, by the way, implement also its parent {\f1\fs20 IDomUserQuery interface};
- The actual implementation class will be {\f1\fs20 TInfraRepoUser} - which will be defined just after;
- The {\i Aggregate/Entity} class is a {\f1\fs20 TUser} kind of object;
- The associated {\f1\fs20 TSQLRest} server will be the one supplied to this class;
- The ORM class, defining the actual SQL table or NoSQL collection which will store the data, is {\f1\fs20 TSQLRecordUser};
- An optional {\f1\fs20 TDDDRepositoryRestManager} instance may be supplied as owner of this factory - but it is not used in most cases.
The {\f1\fs20 AddFilterOrValidate()} method allows to set some @56@ expectations at DDD level. Those rules will be applied before {\f1\fs20 Commit} will take place, without any use of the ORM rules. In the above code, {\f1\fs20 TSynFilterTrim} will remove any space from all text fields of the {\f1\fs20 TUser} instance, and {\f1\fs20 TSynValidateNonVoidText} will ensure that the {\f1\fs20 TUser.LogonName} field will not be {\f1\fs20 ''} - after space trimming. You may consider those rules as the SQL constraints you may be used to. But since they will be defined at DDD level, they will apply on any database back-end, even if it does not support any constraint - e.g. if it is a NoSQL engine, or a third-party persistence service you do not have the hand on.
You will probably want to use those CQRS interfaces, via usual @*IoC@, at {\f1\fs20 TSQLRest} level, just like any @63@:
!var cmd: IDomUserCommand;
!...
!  aServer.Services.Resolve(IDomUserCommand,cmd);
or, for a {\i Query}:
!var qry: IDomUserQuery;
!...
!  aServer.Services.Resolve(IDomUserQuery,qry);
In order to be able to get a {\f1\fs20 IDomUserCommand} or {\f1\fs20 IDomUserQuery} instance from {\f1\fs20 aServer.Services.Resolve()}, you will need to register the {\f1\fs20 TInfraRepoUserFactory} first:
! aServer.ServiceContainer.InjectResolver([TInfraRepoUserFactory.Create(aServer)],true);
or if you want to maintain the factory instance life-time (e.g. to share it with other interface resolvers):
!var factory: TInfraRepoUserFactory;
! ...
!  factory := TInfraRepoUserFactory.Create(aServer);
!  try
!    aServer.ServiceContainer.InjectResolver([factory]);
!  ...
!  finally
!    factory.Free;
!  end;
This single {\f1\fs20 TInfraRepoUserFactory} will allow to implement both {\f1\fs20 IDomUserCommand} and {\f1\fs20 IDomUserQuery} contracts.
Of course, having the ability to let {\f1\fs20 aServer} own the factory, via the {\f1\fs20 InjectResolver([...],true)} parameter, sounds easier to work with.
In practice, for a Client/Server environment, you may write:
!  // Server side
!  RestServer := TSQLRestServerFullMemory.CreateWithOwnModel([TSQLRecordUser]);
!  ...
!  RestServer.ServiceContainer.InjectResolver([TInfraRepoUserFactory.Create(RestServer)],true);
!  RestServer.ServiceDefine(TInfraRepoUser,[IDomUserCommand,IDomUserQuery],sicClientDriven);
!  // now you can use the services on the Server side
!  if RestServer.Services.Resolve(IDomUserCommand,cmd) then
!    ... use cmd
!  if RestServer.Services.Resolve(IDomUserQuery,qry) then
!    ... use qry
!  ...
!  // Client side
!  RestClient := TSQLRestClientURIDll.Create(TSQLModel.Create(...),@URIRequest);
!  ...
!  RestClient.ServiceDefine([IDomUserCommand],sicClientDriven);
!  // now you can use the services on the Client side
!  if RestServer.Services.Resolve(IDomUserCommand,cmd) then
!    ... use cmd
!  if RestServer.Services.Resolve(IDomUserQuery,qry) then
!    ... use qry
Note that {\f1\fs20 InjectResolver()} should be called {\i before} {\f1\fs20 ServiceDefine()}, otherwise the @*IoC@ won't take place as expected, and the {\f1\fs20 TInfraRepoUserFactory} class will be {\f1\fs20 nil}.
The CQRS services should be defined as {\f1\fs20 sicClientDriven} - and not as {\f1\fs20 sicSingle} or {\f1\fs20 sicShared}, since their lifetime is expected to be synchronized by the consumer side, i.e. the interface variable use on the client side.
On the client side, defining {\f1\fs20 IDomUserCommand} is enough to be able to use both {\f1\fs20 IDomUserCommand} and {\f1\fs20 IDomUserQuery} services, but on the server side you will have to explicitly define both interfaces, otherwise the Client/Server contracts won't match and you will not be able to use {\f1\fs20 IDomUserQuery} from the client side.
You could check the {\f1\fs20 TInfraRepoUserFactory.RegressionTests} method, as defined in {\f1\fs20 dddInfraRepoUser.pas}, to find out how such services may be defined and consumed.
:     Implement the CQRS methods
We have defined the factory, and registered the services.\line Now we define the needed methods of {\f1\fs20 IDomUserCommand} and {\f1\fs20 IDomUserQuery} in our custom class:
!type
!  TInfraRepoUser = class(TDDDRepositoryRestCommand,IDomUserCommand,IDomUserQuery)
!  public
!    function SelectByLogonName(const aLogonName: RawUTF8): TCQRSResult;
!    function SelectByEmailValidation(aValidationState: TDomUserEmailValidation): TCQRSResult;
!    function SelectByLastName(const aName: TLastName; aStartWith: boolean): TCQRSResult;
!    function Get(out aAggregate: TUser): TCQRSResult;
!    function GetAll(out aAggregates: TUserObjArray): TCQRSResult;
!    function GetNext(out aAggregate: TUser): TCQRSResult;
!    function Add(const aAggregate: TUser): TCQRSResult;
!    function Update(const aUpdatedAggregate: TUser): TCQRSResult;
!    function HowManyValidatedEmail: integer;
!  end;
Note that we defined the {\f1\fs20 TInfraRepoUser} class as implementing both interface we need, via {\f1\fs20 = class(...,IDomUserCommand,IDomUserQuery}). We need both types to be explicit in the {\f1\fs20 class} type definition, otherwise, @*IoC@ - i.e. {\f1\fs20 aServer.Services.Resolve()} calls - won't work for both.
As you can see, some methods appear to me missing. There is no {\f1\fs20 Commit}, nor {\f1\fs20 Delete} - which are required by {\f1\fs20 IDomUserCommand}. But in fact, those commands are so generic that they are already implemented for you in {\f1\fs20 TDDDRepositoryRestCommand}!
What we need know is to implement those methods, using the internal protected {\f1\fs20 ORM*()} methods inherited by this parent class:
!function TInfraRepoUser.SelectByLogonName(const aLogonName: RawUTF8): TCQRSResult;
!begin
!  result := ORMSelectOne('LogonName=?',[aLogonName],(aLogonName=''));
!end;
!
!function TInfraRepoUser.SelectByEmailValidation(aValidationState: TDomUserEmailValidation): TCQRSResult;
!begin
!  result := ORMSelectAll('EmailValidated=?',[ord(aValidationState)]);
!end;
!
!function TInfraRepoUser.SelectByLastName(const aName: TLastName; aStartWith: boolean): TCQRSResult;
!begin
!  if aStartWith then
!    result := ORMSelectAll('Name_Last LIKE ?',[aName+'%'],(aName='')) else
!    result := ORMSelectAll('Name_Last=?',[aName],(aName=''));
!end;
!
!function TInfraRepoUser.Get(out aAggregate: TUser): TCQRSResult;
!begin
!  result := ORMGetAggregate(aAggregate);
!end;
!
!function TInfraRepoUser.GetAll(out aAggregates: TUserObjArray): TCQRSResult;
!begin
!  result := ORMGetAllAggregates(aAggregates);
!end;
!
!function TInfraRepoUser.GetNext(out aAggregate: TUser): TCQRSResult;
!begin
!  result := ORMGetNextAggregate(aAggregate);
!end;
!
!function TInfraRepoUser.Add(const aAggregate: TUser): TCQRSResult;
!begin
!  result := ORMAdd(aAggregate);
!end;
!
!function TInfraRepoUser.Update(const aUpdatedAggregate: TUser): TCQRSResult;
!begin
!  result := ORMUpdate(aUpdatedAggregate);
!end;
!
!function TInfraRepoUser.HowManyValidatedEmail: integer;
!begin
!  if ORMSelectCount('EmailValidated=%',[ord(evValidated)],[],result)<>cqrsSuccess then
!    result := 0;
!end;
Almost everything is already defined at {\f1\fs20 TDDDRepositoryRestCommand} level. Our {\f1\fs20 TInfraRepoUser} class, implementing a full CQRS service, fully abstracted from the ORM, is implemented by a few internal {\f1\fs20 ORM*()} method calls.
All the error handling, including server-side {\f1\fs20 exception} catching, and conversion into {\f1\fs20 TCQRSResult} / {\f1\fs20 ICQRSService.GetLastErrorInfo} content, is already implemented in {\f1\fs20 TDDDRepositoryRestCommand}.
All the data access via the {\f1\fs20 TSQLRecordUser} REST persistence layer, with any @56@ defined rule, is also incorporated in {\f1\fs20 TDDDRepositoryRestCommand}. The conversion to/from {\f1\fs20 TUser} properties has been optimized, so that fields will be moved {\i by reference}, with no memory allocation nor content modification, for best performance and data safety. The type mapping specified by {\f1\fs20 TInfraRepoUserFactory.Create} is enough to make the whole process as automated as possible.
In fact, our {\f1\fs20 TInfraRepoUser} {\f1\fs20 class} is just a thin wrapper forcing use of strong typing in its methods parameters (i.e. using {\f1\fs20 TUser}/{\f1\fs20 TUserObjArray} whereas the {\f1\fs20 ORM*()} methods are more relaxed about actual typing), and ensuring that the ORM specificities are followed as expected, e.g. a search against the {\f1\fs20 TUser.Name.Last} DDD field will use the {\f1\fs20 TSQLRecordUser.Name_Last} ORM column, with the proper {\f1\fs20 LIKE} operator.
Internally, {\f1\fs20 TDDDRepositoryRestCommand.ORMPrepareForCommit} will call all DDD and ORM {\f1\fs20 TSynFilter} and {\f1\fs20 TSynValidate} rules, as previously defined. It sounds indeed like a real advantage not to wait until the database layer is reached, to have those constraints verified. The sooner an error is notified, the better - especially in a complex @*SOA@ system.
Under the hood, {\f1\fs20 TDDDRepositoryRestCommand} will define a {\f1\fs20 TSqlRestBatch} - see @28@ - for storing all write commands in memory (as JSON) - e.g. {\f1\fs20 cmd.Add} - and will send them to the database engine, with optimized SQL or NoSQL statements, only when {\f1\fs20 cmd.Commit} will be executed.
:   Isolate your Domain using DTOs
DDD's {\i @*DTO@} may also be defined as {\f1\fs20 record}, and directly serialized as JSON via text-based serialization. Don't be afraid of writing some translation layers between {\f1\fs20 TSQLRecord} and DTO records or, more generally, between your {\i Application layer} and your {\i Presentation layer}. It will be very fast, on the server side. If your service interfaces are cleaner, do not hesitate.
But defining {\i DTO} types, just for uncoupling, may become time consuming. If you start writing a lot of wrapping code, forget about it, and expose your Domain {\i Value Objects} or even your {\i Entities}, as stated above. Or automate the wrapper coding, using RTTI and code generators. You have to weight the PROs and the CONs, like always... And never forget to write proper unit testing of this marshalling code, since it may induce some unexpected issues.
If you expect your DDD's objects to be {\i schema-less} or with an evolving structure (e.g. for {\i DTO}), depending on each context, you may benefit of not using a fixed {\f1\fs20 type} like {\f1\fs20 class} or {\f1\fs20 record}, but use @80@. This kind of {\f1\fs20 variant} will be serialized as JSON, and allow @*late-binding@ access to its properties (for {\i object} documents) or items (for {\i array} documents). In the context of interface-based services, using {\i per-reference} option at creation (i.e. {\f1\fs20 _ObjFast() _ArrFast() _JsonFast() _JsonFmtFast()} functions) does make sense, in order to spare the server resources.
:  Defining services
In practice, {\i mORMot}'s Client-Server architecture may be used as such:
- {\i @*Service@s via methods} - see @49@ - can be used to publish methods corresponding to your aggregate roots defined as {\f1\fs20 TSQLRecord}.\line This will make it pretty @*REST@ful compatible.
- {\i Services via interfaces} - see @63@ - can be used to publish all your processes.\line Dedicated factories can be used on both Client and Server side, to define your repositories and/or domain operations.
@49@ may be preferred if you expect your service to be consumed in a truly @*REST@ful way. But since in DDD you should better protect your {\i Domain} via a dedicated @*Adapter@ layer, such compatibility should be an implementation smell. In practice, @63@ will offer better integration and automation of its process, e.g. parameter type validation (with @*JSON@ marshalling), @*session@ handling, {\f1\fs20 interface}-level @*multi-thread@ing and @*security@ abilities, @*log@ging, ability to be emulated via @166@, and - last but not least - @173@.
:  Event-Driven Design
{\i @**Event-Driven@} could be implemented in {\i mORMot} by at least two ways:
- Using {\f1\fs20 interface} callbacks of the framework @63@;
- Storing the system {\i states} in a table, and let @153@ generate the events.
Both ways have their own benefits and drawbacks, and you may pick up the one which match your particular use case. The first may be more easy to implement and versatile to use, but the second will work with {\i off-line} periods, and
:172   Events as Callbacks
DDD's {\i @**Events@} could easily be implemented as @149@, when an {\f1\fs20 interface} callback is defined as @154@. In this case, the {\f1\fs20 interface} type will define the various DDD events, ready to be notified and propagated in real-time across the whole system.
An application layer may provide a specific callback to the domain, which will push the notification as a regular Delphi call, but in fact transmitted via {\i @*WebSockets@} from the corresponding Domain Service to the right application layer. The current implementation relies on {\i WebSockets} for remote access, but other protocols may be available in the future, since {\f1\fs20 interface} parameters callbacks may be implemented by any actual transmission {\f1\fs20 class}.
No need to encapsulate your events within a dedicated message class (as most @*Event-Driven@ implementations require), or pollute your {\i Domain} code to follow a fixed protocol expectations: just run a notification method corresponding to the event, and you are done - all subscribers will be notified.
No need to put in production a @*Message bus@, or a centralized system. Using callbacks, you will your outer layers (e.g. {\i Application} or {\i Presentation} layers) be cleanly notified by the Domain Services, without any waste of resource, and without potential bottleneck. Each node of your system will communicate directly with its subscriber, from a pure {\f1\fs20 interface} method call, as if it was a local process. See @173@ for implementation details.
In practice, the callbacks may be propagated from the {\i Domain} layer to the {\i Application} or {\i Presentation} layers, which may also have their own callbacks definitions, using not {\i Domain} objects, but their own @*DTO@s. Marshalling an event will be as easy as writing a class implementing an {\f1\fs20 I*Callback} interface as defined in the {\i Domain}, translating its parameters into the DTO types are defined for the outer {\i Application} or {\i Presentation} services.
On the server side, you may even define the callbacks in the very same process, without the {\i WebSockets} overhead, but calling directly the {\i Domain} services, whose {\f1\fs20 interface} type will be defined at {\i Domain} level, but the {\f1\fs20 class} type implemented at {\i Infrastructure} level:
- The application layer may be able to run directly the {\i Domain} code in its own service/daemon, calling the actual implementation at {\i Infrastructure} level, with a single straight {\i WebSockets} transmission - the DDD's {\i @*Adapter@s} types being pure Delphi classes, running in process, with no overhead.
- Or, you may gather {\i Domain} Services in some specific stand-alone daemons, which may be able to cache the events, and/or centralize some process - as a benefit, it may help those services be truly {\i @*stateless@}, so the {\i Application} Layer may become redundant for better scaling.
Using {\f1\fs20 interface} values and call their methods is a natural way of writing callbacks in {\i Delphi} code, very close to the VCL/RAD Events you may be used to, but with the benefit of the abstraction of @46@, especially @47@. If your need is to react in real time to some change of the system, they are probably the preferred way.
:171   Event Sourcing via Event Oriented Databases
Another new, and popular DDD's {\i Events} implementation pattern is to define some kind of event persistence, which will be used as {\i @*Event Sourcing@}. Here, we won't rely on explicit messages to transmit the events (as we just proposed via asynchronous {\f1\fs20 interface} callbacks), but we will use some {\i state} storage in an {\i @**Event Oriented Persistence@}, then let subscribers by notified for each state change.
In this pattern, there are not directly any kind of {\i Event} defined. The state of the {\i Domain} is stored somewhere, then any change of state should be notified to whom has interest for it. Obviously, one potential easy implementation may be via @153@, as proposed by the framework.
The {\i Domain} services - see e.g. @102@ or if you @167@ - may modify a dedicated {\f1\fs20 TSQLRecord} table, which will contain only a small part of the state of the model. For instance, its @26@ may store only a few evolving values, like the latest order placed, or the price of an item, or the connection state of a peripheral. The main point is to restrict the data stored to its minimum, e.g. this evolving value and the name (or ID) of the object.
Thanks to the framework @153@, any client process or service, via its own {\i Slave} copy of this {\f1\fs20 TSQLRecord} {\i State} storage, will be notified asynchronously. This notification will reflect each change of state, and will let the consumer react as expected. One {\f1\fs20 OnNotify} event is available, to track each individual change of state, as specified as parameter to {\f1\fs20 TSQLRestServer.RecordVersionSynchronizeSlaveStart}.
When using @172@, you may miss some events: if the consumer service is off-line, there won't be any event notified. It may be as expected, but may be a huge issue in some cases. On the other hand, the {\i Event Oriented Persistence} model will allow the consumers to be safely off-line at some time. Each ORM {\i Slave} will have its own copy of the data, then will be able to retrieve all the missed changes of state, when it goes on-line.
This implementation pattern is in fact the base of any true {\i Event Sourcing} process. Following this DDD pattern, each node of the system should {\i store} the data it needs. The system nodes won't ask for a given information (e.g. "What is the current temperature?"), but will be notified of each temperature change, then store the value, and being able to propagate any incoming events with almost no dependency. The main benefit is that you could add some node to the system, without any prior knowledge of what is already there. Such {\i Events-driven Architecture} (EDA) or {\i Domain Event Driven Service Oriented Architecture} (D-EDA) may be complex to maintain and debug, once they reach a given size. For instance, some unexpected {\i Event Cascade} may happen, when you get a sequence of events triggering other events: you may induce an infinite rebound in the whole system. As a consequence, a "pure Event Driven" system will probably be a wrong idea. {\i Event Sourcing} may be introduced for some part of your {\i Domain}, where it does make better sense. See @http://martinfowler.com/eaaDev/EventCollaboration.html for more material.
As a side benefit, scaling of the whole system may be increased by this pattern. Each {\i Event State} storage may be seen as a safe cache of the system state, in the bounded context of a given set of values. When your business logic wonder about this particular state, it may ask this dedicated service, leveraging the main database. You may even consider storing the whole state history in a dedicated @85@ storage, without impacting the whole system.
:  Building a Clean architecture
A common DDD architecture is expressed as in the following model, which may look like a regular @7@ design at first, but should be implemented as a @%%mORMotDesignOnion@:
|%30%50
|\b Layer|Description\b0
|Presentation|MVC UI generation and reporting
|Application|Services and high-level adapters
|Domain Model|Where business logic remains
|Data persistence|ORM and external services
|Cross-Cutting|Horizontal aspects shared by other layers
|%
{\i Physically}, it involves a common {\i @*n-Tier@} representation splitting the classical {\i Logic Tier} into two layers, i.e. {\i Application layer} and {\i Domain Model layer}. At {\i logical} level, DDD will try to uncouple the {\i Domain Model layer} from other layers, so the code itself will rely on {\f1\fs20 interface}s and {\i dependency injection} to let the core {\i Domain} focus on the business logic, not on implementation details (e.g. persistence or communication).
The @*REST@ful @*SOA@ components of our {\i Synopse mORMot framework} can therefore define such an Architecture:
\graph mORMotDesign0 Clean Domain-Oriented Architecture of mORMot
subgraph cluster1 {
\AJAX\REST Client
\UI\REST Client
\UI\i18n
\Reporting\i18n
\Reporting\REST Client
label="Presentation";
}
subgraph cluster2 {
"REST Server";
label="Application";
}
\REST Client\REST Server\HTTP 1.1
subgraph cluster3 {
\ORM\SQLite3
\ORM\External DB
\SQLite3\External DB
label="Data persistence";
}
subgraph cluster4 {
\REST Server\Services
\Services\Value Objects
\Services\Entities
\Services\Aggregates
\REST Server\ORM
\Entities\ORM
\Aggregates\ORM
label="Domain\nModel";
}
subgraph cluster5 {
label="Cross-Cutting";
"Logging";
\UI\Filtering¤Validation
\REST Server\Security¤Sessions
\REST Client\Cache
\ORM\Cache
\Services\Tests¤Mocks/Stubs
\Services\Security¤Sessions
\Services\Filtering¤Validation
}
=Services=Services¤(interface-based)
\
As we already stated, the main point of this {\i Clean Architecture} is to control coupling, and isolate the {\i Domain} core from the outer layers. In {\i Delphi}, unit dependencies (as displayed e.g. by our {\i @*SynProject@} tool) will be a good testimony of proper objects uncoupling: in the units defining your domain, you may split it between {\i Domain Model} and {\i Domain Services} (the 2nd using the first, and not vice-versa), and you should {\i never} have any dependency to a particular DB unit, just to the framework's core units, i.e. {\f1\fs20 SynCommons.pas} and {\f1\fs20 mORMot.pas}. @62@ - via @63@ or at @*ORM@ initialization level - will ensure that your code is uncoupled from any low-level technical dependency. It will also allow proper testing of your application workflows, e.g. stubbing the database if necessary.
In fact, since a @17@ tends to ensure that services comprise unassociated, loosely coupled units of functionality that have no calls to each other embedded in them, we may define two levels of services, implemented by two {\f1\fs20 interface} factories, using their own @*hosting@ and communication:
- One set of @*service@s at {\i Application layer}, to define the uncoupled contracts available from Client applications;
- One set of services at {\i Domain Model layer}, which will allow all involved domains to communicate with each other, without exposing it to the remote clients.
Therefore, those layers could be also implemented as such:
\graph mORMotDesign0b Alternate Domain-Oriented Architecture of mORMot
rankdir=LR;
subgraph cluster2 {
\Application¤REST Server\Application¤Services
\Application¤Services\Domain¤REST Client
label="Application";
}
\Application¤REST Client\Application¤REST Server\HTTP 1.1
subgraph cluster4 {
\Domain¤REST Client\Domain¤REST Server\Fast¤RPC
\Domain¤REST Server\Domain¤Services
label="Domain Model";
}
\Domain¤REST Server\ORM
\Domain¤Services\ORM
\
In order to provide the better scaling of the server side, @*cache@ can be easily implemented at every level, and hosting can be tuned in order to provide the best response time possible: one central server, several dedicated servers for application, domain and persistence layers...
Due to the @*SOLID@ design of {\i mORMot} - see @47@ - you can use as many Client-Server services layers as needed in the same architecture (i.e. a Server can be a Client of other processes), in order to fit your project needs, and let it evolve from the simplest architecture to a full scalable {\i Domain-Driven} design.
:12Testing and logging
%cartoon07.png
Since we covered most architectural and technical aspects of the framework, it is time to put the last missing bricks to the building, meaning testing and logging.
\page
: Automated testing
You know that @**test@ing is (almost) everything if you want to avoid regression problems in your application.
How can you be confident that any change made to your software code won't create any error in other part of the software?
Automated unit testing is a good candidate for avoiding any serious regression.
And even better, testing-driven coding can be encouraged:
- Write a void implementation of a feature, that is code the interface with no implementation;
- Write a test code;
- Launch the test - it must fail;
- Implement the feature;
- Launch the test - it must pass;
- Add some features, and repeat all previous tests every time you add a new feature.
It could sounds like a waste of time, but such coding improve your code quality a lot, and, at least, it help you write and optimize every implementation feature.
The framework has been implemented using this approach, and provide all the tools to write tests. In addition to what other {\i Delphi} frameworks offer (e.g. {\i DUnit / DUnitX}), the {\f1\fs20 SynTests.pas} unit is very much integrated with other elements of the framework (like logging), is cross-platform and cross-compiler, and provides a complete {\i stubbing / mocking} mechanism to cover @62@.
:  Involved classes in Unitary testing
The @!TSynTest,TSynTestCase,TSynTests!Lib\SynTests.pas@ unit defines two classes (both inheriting from {\f1\fs20 TSynTest}), implementing a complete Unitary testing mechanism similar to {\i DUnit}, with less code overhead, and direct interface with the framework units and requirements (@*UTF-8@ ready, code compilation from {\i Delphi} 6 up to the latest available {\i Delphi} version and FPC, no external dependency).
The following diagram defines this class hierarchy:
\graph HierTSynTest TSynTest classes hierarchy
\TSynTests\TSynTest
\TSynTestCase\TSynTest
\
The main usable class types are:
- {\f1\fs20 TSynTestCase}, which is a class implementing a test case: individual tests are written in the published methods of this class;
- {\f1\fs20 TSynTests}, which is used to run a suit of test cases, as defined with the previous class.
In order to define tests, some {\f1\fs20 TSynTestCase} children must be defined, and will be launched by a {\f1\fs20 TSynTests} instance to perform all the tests. A text report is created on the current console, providing statistics and Pass/Fail.
:  First steps in testing
Here are the functions we want to test:
!function Add(A,B: double): Double; overload;
!begin
!  result := A+B;
!end;
!
!function Add(A,B: integer): integer; overload;
!begin
!  result := A+B;
!end;
!
!function Multiply(A,B: double): Double; overload;
!begin
!  result := A*B;
!end;
!
!function Multiply(A,B: integer): integer; overload;
!begin
!  result := A*B;
!end;
So we create three classes one for the whole test suit, one for testing addition, one for testing multiplication:
!type
!  TTestNumbersAdding = class(TSynTestCase)
!  published
!    procedure TestIntegerAdd;
!    procedure TestDoubleAdd;
!  end;
!
!  TTestNumbersMultiplying = class(TSynTestCase)
!  published
!    procedure TestIntegerMultiply;
!    procedure TestDoubleMultiply;
!  end;
!
!  TTestSuit = class(TSynTests)
!  published
!    procedure MyTestSuit;
!  end;
The trick is to create published methods, each containing some tests to process.
Here is how one of these test methods are implemented (I let you guess the others):
!procedure TTestNumbersAdding.TestDoubleAdd;
!var A,B: double;
!    i: integer;
!begin
!  for i := 1 to 1000 do
!  begin
!    A := Random;
!    B := Random;
!    CheckSame(A+B,Adding(A,B));
!  end;
!end;
The {\f1\fs20 CheckSame()} is necessary because of floating-point precision problem, we can't trust plain = operator (i.e. {\f1\fs20 Check(A+B=Adding(A,B))} will fail because of rounding problems).
And here is the test case implementation:
!procedure TTestSuit.MyTestSuit;
!begin
!  AddCase([TTestNumbersAdding,TTestNumbersMultiplying]);
!end;
And the main program (this {\f1\fs20 .dpr} is expected to be available as a console program):
!  with TTestSuit.Create do
!  try
!    ToConsole := @Output; // so we will see something on screen
!    Run;
!    readln;
!  finally
!    Free;
!  end;
Just run this program, and you'll get:
$
$   Suit
$  ------
$
$
$1. My test suit
$
$ 1.1. Numbers adding:
$  - Test integer add: 1,000 assertions passed  92us
$  - Test double add: 1,000 assertions passed  125us
$  Total failed: 0 / 2,000  - Numbers adding PASSED  360us
$
$ 1.2. Numbers multiplying:
$  - Test integer multiply: 1,000 assertions passed  73us
$  - Test double multiply: 1,000 assertions passed  117us
$  Total failed: 0 / 2,000  - Numbers multiplying PASSED  324us
$
$
$Generated with: Delphi 7 compiler
$
$Time elapsed for all tests: 1.51ms
$Tests performed at 25/03/2014 10:59:33
$
$Total assertions failed for all test suits:  0 / 4,000
$! All tests passed successfully.
You can see that all text on screen was created by "UnCamelCasing" the method names (thanks to our good old @*Camel@), and that the test suit just follows the order defined when registering the classes. Each method has its own timing, which is pretty convenient to track performance regressions.
This test program has been uploaded in the {\f1\fs20 SQLite3\\Sample\\07 - SynTest} folder of the Source Code Repository.
:  Framework test coverage
The @SAD-DI-2.2.2@ defines all classes released with the framework source code, which covers all core aspects of the framework. Global testing coverage is good, excellent for core components (more than 25,000,000 individual checks are performed for revision 1.18), but there is still some User-Interface related tests to be written.
Before any release all unitary regression tests are performed with the following compilers:
- {\i @*Delphi@} 5 (for a limited scope, including {\f1\fs20 SynCommons.pas}, {\f1\fs20 SynLog.pas}, {\f1\fs20 SynTests.pas}, {\f1\fs20 SynCryto.pas}, {\f1\fs20 SynEcc.pas}, {\f1\fs20 SynPdf.pas} and {\f1\fs20 SynDB.pas});
- {\i Delphi} 6;
- {\i Delphi} 7, with and without our Enhanced Run Time Library;
- {\i Delphi} 2007;
- {\i Delphi} 2010 (we assume that if it works with {\i Delphi} 2010, it will work with {\i Delphi} 2009, with the exception of {\f1\fs20 generic} compilation);
- {\i Delphi} XE4;
- {\i Delphi} XE7;
- {\i Delphi} XE8;
- {\i Delphi} 10.3 Rio;
- {\i Delphi} 10.4 Sidney;
- {\i @*CrossKylix@} 3.0;
- {\i @*FPC@} 3.x - preferred is {\i 3.2 fixes}.
Target platforms are {\i Win32} and {\i Win64} for {\i Delphi} and {\i FPC}, plus {\i Linux 32/64} for {\i FPC} and {\i CrossKylix}.
Then all sample source code (including the {\i Main Demo} and {\f1\fs20 @*SynDBExplorer@} sophisticated tools) are compiled, and user-level testing is performed against those applications.
You can find in the {\f1\fs20 compil.bat} and {\f1\fs20 compilpil.bat} files of our source code repository how incremental builds and tests are performed.
\page
:16 Enhanced logging
A @**log@ging mechanism is integrated with cross-cutting features of the framework. It includes stack trace exception and such, just like {\i MadExcept}, using {\f1\fs20 .map} file content to retrieve debugging information from the source code.
Here are some of its features:
- Logging with a {\i set} of levels, not only a level scale;
- Fast, low execution overhead;
- Can load {\f1\fs20 .map} file symbols to be displayed in logging (i.e. source code file name and line numbers are logged instead of a hexadecimal value);
- Compression of {\f1\fs20 .map} into binary {\f1\fs20 .mab} (900 KB -> 70 KB);
- Inclusion of the {\f1\fs20 .map/.mab} into the {\f1\fs20 .exe}, with very slow size increase;
- Exception logging ({\i Delphi} or low-level exceptions) with unit names and line numbers;
- Optional stack trace with units and line numbers;
- Methods or procedure recursive tracing, with {\i Enter} and {\i auto-Leave} (using a fake {\f1\fs20 interface} instance);
- High resolution time stamps, for customer-side profiling of the application execution;
- Set / enumerates / {\f1\fs20 TList} / {\f1\fs20 TPersistent} / {\f1\fs20 TObjectList} / dynamic array JSON serialization;
- Per-thread or global logging;
- Optional multiple log files on the same process;
- Optional rotation when main log reaches a specified size, with compression of the rotated logs;
- Integrated log archival (in {\f1\fs20 .zip} or any other format, including our {\f1\fs20 .synlz});
- Optional colored echo to a console window, for interactive debugging;
- Fast log viewer tool available, including thread filtering and customer-side execution profiling;
- Optional remote logging via HTTP - the log viewer can be used as server;
- Optional events transmission to a UDP {\f1\fs20 syslog} server.
:  Setup logging
Logging is defined mainly by a per-class approach. You usually define your logging expectations by using a {\f1\fs20 TSynLog} class, and setting its {\f1\fs20 Family} property. Note that it is perfectly feasible to use you own {\f1\fs20 TSynLog} class instance, with its own {\f1\fs20 TSynLog} family settings, injected at the {\f1\fs20 constructor} level; but in {\i mORMot}, we usually use the per-class approach, via {\f1\fs20 TSynLog}, {\f1\fs20 TSQLLog}, {\f1\fs20 SynDBLog} and {\f1\fs20 SQLite3Log} - see @73@.
For sample code (and the associated log viewer tool), see "{\i 11 - Exception logging}" folder in "{\i Sqlite3\\Samples}".
In short, you can add logging to your program, just by using the {\f1\fs20 TSynLog} class, as such:
!  TSynLog.Add.Log(sllInfo,Stats.DebugMessage);
This line will log the {\f1\fs20 Stats.DebugMessage} text, with a {\f1\fs20 sllInfo} notification level. See the description of all {\f1\fs20 Log()} overloaded methods of the {\f1\fs20 ISynLog} interface, to find out how your project can easily log events.
First of all, you need to define your logging setup via code:
!  with TSynLog.Family do begin
!    Level := LOG_VERBOSE;
!    //Level := [sllException,sllExceptionOS];
!    //HighResolutionTimestamp := true;
!    //AutoFlushTimeOut := 5;
!    OnArchive := EventArchiveSynLZ;
!    //OnArchive := EventArchiveZip;
!    ArchiveAfterDays := 1; // archive after one day
!  end;
The main setting here is {\f1\fs20 TSynLog.Family.Level := ...} which defines which levels are to be logged. That is, if {\f1\fs20 sllInfo} is part of {\f1\fs20 TSynLog.Family.Level}, any {\f1\fs20 TSynLog.Add.Log(sllInfo,...)} command will log the corresponding content - otherwise, it will be a no-operation. {\f1\fs20 LOG_VERBOSE} is a constant setting all levels at once.
You have several debugging levels available, and even 4 custom types:
!  TSynLogInfo = (
!    sllNone, sllInfo, sllDebug, sllTrace, sllWarning, sllError,
!    sllEnter, sllLeave,
!    sllLastError, sllException, sllExceptionOS, sllMemory, sllStackTrace,
!    sllFail, sllSQL, sllCache, sllResult, sllDB, sllHTTP, sllClient, sllServer,
!    sllServiceCall, sllServiceReturn, sllUserAuth,
!    sllCustom1, sllCustom2, sllCustom3, sllCustom4, sllNewRun);
Here are the purpose of each logging level:
- {\f1\fs20 sllInfo} will log general information events;
- {\f1\fs20 sllDebug} will log detailed debugging information;
- {\f1\fs20 sllTrace} will log low-level step by step debugging information;
- {\f1\fs20 sllWarning} will log unexpected values (not an error);
- {\f1\fs20 sllError} will log errors;
- {\f1\fs20 sllEnter} will log every method start;
- {\f1\fs20 sllLeave} will log every method quit;
- {\f1\fs20 sllLastError} will log the {\f1\fs20 GetLastError} OS message;
- {\f1\fs20 sllException} will log all exception raised - available since Windows XP;
- {\f1\fs20 sllExceptionOS} will log all OS low-level exceptions ({\f1\fs20 EDivByZero, ERangeError, EAccessViolation}...);
- {\f1\fs20 sllMemory} will log memory statistics;
- {\f1\fs20 sllStackTrace} will log caller's stack trace (it is by default part of {\f1\fs20 TSynLogFamily. LevelStackTrace} like {\f1\fs20 sllError, sllException, sllExceptionOS, sllLastError} and {\f1\fs20 sllFail});
- {\f1\fs20 sllFail} was defined for {\f1\fs20 TSynTestsLogged. Failed} method, and can be used to log some customer-side assertions (may be notifications, not errors);
- {\f1\fs20 sllSQL} is dedicated to trace the SQL statements;
- {\f1\fs20 sllCache} should be used to trace any internal caching mechanism (it is used for instance by our SQL statement caching);
- {\f1\fs20 sllResult} could trace the SQL results, JSON encoded;
- {\f1\fs20 sllDB} is dedicated to trace low-level database engine features;
- {\f1\fs20 sllHTTP} could be used to trace HTTP process;
- {\f1\fs20 sllClient/sllServer} could be used to trace some Client or Server process;
- {\f1\fs20 sllServiceCall/sllServiceReturn} to trace some remote service or library;
- {\f1\fs20 sllUserAuth} to trace user authentication (e.g. for individual requests);
- {\f1\fs20 sllCustom1..sllCustom4} items can be used for any purpose by your programs;
- {\f1\fs20 sllNewRun} will be written when a process re-opens a rotated log.
Logging is not using directly a {\f1\fs20 TSynLogInfo} level, but the following {\f1\fs20 set}:
!  /// used to define a logging level
!  // - i.e. a combination of none or several logging event
!  // - e.g. use LOG_VERBOSE constant to log all events
!  TSynLogInfos = set of TSynLogInfo;
Most logging tools in the wild use a level scale, i.e. with a hierarchy, excluding the lower levels when one is selected.
Our logging classes use a {\i set}, and not directly a particular level, so you are able to select which exact events are worth recording. In practice, we found this pattern to make a lot of sense and to be much more efficient for support.
:  Call trace
The logging mechanism can be used to trace recursive calls. It can use an interface-based mechanism to log when you enter and leave any method:
!procedure TMyDB.SQLExecute(const SQL: RawUTF8);
!var ILog: ISynLog;
!begin
!  ILog := TSynLogDB.Enter(self,'SQLExecute');
!  // do some stuff
!  ILog.Log(sllInfo,'SQL=%',[SQL]);
!end; // when you leave the method, it will write the corresponding event to the log
It will be logged as such:
$20110325 19325801  +    MyDBUnit.TMyDB(004E11F4).SQLExecute
$20110325 19325801 info   SQL=SELECT * FROM Table;
$20110325 19325801  -
Note that by default you have human-readable {\i time and date} written to the log, but it is also possible to replace this timing with {\i high-resolution timestamps}. With this, you'll be able to profile your application with data coming from the customer side, on its real computer. Via the {\f1\fs20 Enter} method (and its {\i auto-Leave} feature), you have all information needed for this.
:  Including symbol definitions
In the above logging content, the method name is set in the code (as {\f1\fs20 'SQLExecute'}). But if the logger class is able to find a {\f1\fs20 .map} file associated to the {\f1\fs20 .exe}, the logging mechanism is able to read this symbol information, and write the exact line number of the event.
By default, the {\f1\fs20 .map} file information is not generated by the compiler. To force its creation, you must ensure the {\f1\fs20 \{$D+\}} compiler directive is set in every unit (which is the case by default, unless you set {\f1\fs20 \{$D-\}} in the source), and the "{\i Detailed Map File}" option selected in the {\i Project > Options > Linker} page of the {\i Delphi} IDE.
In the following log entries, you'll see both high-resolution time stamp, and the entering and leaving of a {\f1\fs20 TTestCompression.TestLog} method traced with no additional code (with accurate line numbers, extracted from the {\f1\fs20 .map} content):
$0000000000000B56  +    TTestCompression(00AB3570).000E6C79 SynSelfTests.TTestCompression.TestLog (376)
$0000000000001785  -
There is already a dedicated {\f1\fs20 TSynLogFile} class able to read the {\f1\fs20 .log} file, and recognize its content.
The first time the {\f1\fs20 .map} file is read, a {\f1\fs20 .mab} file is created, and will contain all symbol information needed. You can send the {\f1\fs20 .mab} file with the {\f1\fs20 .exe} to your client, or even embed its content to the {\f1\fs20 .exe} (see the {\f1\fs20 Map2Mab.dpr} sample file located in the {\f1\fs20 Samples\\11 - Exception logging\\} folder).
This {\f1\fs20 .mab} file is very optimized: for instance, a {\f1\fs20 .map} of 927,984 bytes compresses into a 71,943 {\f1\fs20 .mab} file.
:  Exception handling
Of course, this @*log@ging mechanism is able to intercept the raise of exceptions, including the worse (e.g. {\f1\fs20 EAccessViolation}), to be logged automatically in the log file, as such:
$000000000000090B EXCOS EAccessViolation (C0000005) at 000E9C7A SynSelfTests.Proc1 (785)  stack trace 000E9D51 SynSelfTests.Proc2 (801) 000E9CC1 SynSelfTests.Proc1 (790) 000E9D51 SynSelfTests.Proc2 (801) 000E9CC1 SynSelfTests.Proc1 (790) 000E9D51 SynSelfTests.Proc2 (801) 000E9CC1 SynSelfTests.Proc1 (790) 000E9D51 SynSelfTests.Proc2 (801) 000E9CC1 SynSelfTests.Proc1 (790) 000E9D51 SynSelfTests.Proc2 (801) 000E9CC1 SynSelfTests.Proc1 (790) 000E9E2E SynSelfTests.TestsLog (818) 000EA0FB SynSelfTests (853) 00003BF4 System.InitUnits 00003C5B System.@StartExe 000064AB SysInit.@InitExe 000EA3EC TestSQL3 (153)
The {\f1\fs20 TSynLogInfo} logging level makes a difference between high-level {\i Delphi} exceptions ({\f1\fs20 sllException}) and lowest-level OS exceptions ({\f1\fs20 sllExceptionOS}) like {\f1\fs20 EAccessViolation}.
For instance, if you add to your program:
!uses
!  SynLog;
!(...)
!  TSynLog.Family.Level := [sllExceptionOS];
all OS exceptions (excluding pure {\i Delphi} exception like {\f1\fs20 EConvertError} and such) will be logged to a separated log file.
!TSynLog.Family.Level := [sllException,sllExceptionOS];
will trace also {\i Delphi} exceptions, for instance.
You can specify some {\f1\fs20 Exception} class to be ignored, by adding them to {\f1\fs20 Family.ExceptionIgnore} internal list. It could make sense to add this setting, if your code often triggers some non-breaking exceptions, e.g. with {\f1\fs20 StrToInt()}:
!  TSynLog.Family.ExceptionIgnore.Add(EConvertError);
If your {\i Delphi} code executes some {\i .Net} managed code (e.g. exposed via some COM wrapper components), the unit is able to recognize most un-handled {\i .Net} exceptions, and log them with their original {\f1\fs20 C#} class name (for instance, {\f1\fs20 EOleSysError 80004003} will be recorded as a much more user-friendly "{\f1\fs20 [.NET/CLR unhandled ArgumentNullException]}" message.
You can set the following global variable to assign a customized callback, and be able to customize the logging content associated to any exception:
!type
!  /// global hook callback to customize exceptions logged by TSynLog
!  // - should return FALSE if Context.EAddr and Stack trace is to be appended
!  TSynLogExceptionToStr = function(WR: TTextWriter; const Context: TSynLogExceptionContext): boolean;
!
!var
!  /// allow to customize the Exception logging message
!  TSynLogExceptionToStrCustom: TSynLogExceptionToStr = nil;
The {\f1\fs20 Context: TSynLogExceptionContext} content is to be used to append some text to the specified {\f1\fs20 TTextWriter} instance.
An easier possibility is to inherit your custom exception class from {\f1\fs20 ESynException}, and override its unique virtual method:
!  /// generic parent class of all custom Exception types of this unit
!  ESynException = class(Exception)
!  public
!    /// can be used to customize how the exception is logged
!    // - this default implementation will call the DefaultSynLogExceptionToStr()
!    // callback or TSynLogExceptionToStrCustom, if defined
!    // - override this method to provide a custom logging content
!    // - should return TRUE if Context.EAddr and Stack trace is not to be
!    // written (i.e. as for any TSynLogExceptionToStr callback)
!!    function CustomLog(WR: TTextWriter; const Context: TSynLogExceptionContext): boolean; virtual;
!  end;
See {\f1\fs20 TSynLogExceptionContext} to check the execution context, and the implementation of the {\f1\fs20 function DefaultSynLogExceptionToStr()} function.
:  Serialization
{\i @*dynamic array@s} can also be serialized as @*JSON@ in the log on request, via the default {\f1\fs20 TSynLog} class, as defined in {\f1\fs20 SynLog.pas} unit - see @48@.
The {\f1\fs20 TSQLLog} class (using the enhanced @*RTTI@ methods defined in {\f1\fs20 mORMot.pas} unit) is even able to serialize {\f1\fs20 @*TSQLRecord@, @*TPersistent@, TList} and {\f1\fs20 @*TCollection@} instances as JSON, or any other class instance, after call to {\f1\fs20 TJSONSerializer. @*RegisterCustomSerializer@}.
For instance, the following code:
!procedure TestPeopleProc;
!var People: TSQLRecordPeople;
!    Log: ISynLog;
!begin
!  Log := TSQLLog.Enter;
!  People := TSQLRecordPeople.Create;
!  try
!    People.ID := 16;
!    People.FirstName := 'Louis';
!    People.LastName := 'Croivébaton';
!    People.YearOfBirth := 1754;
!    People.YearOfDeath := 1793;
!    Log.Log(sllInfo,People);
!  finally
!    People.Free;
!  end;
!end;
will result in the following log content:
$0000000000001172  +    000E9F67 SynSelfTests.TestPeopleProc (784)
$000000000000171B info      {"TSQLRecordPeople(00AB92E0)":{"ID":16,"FirstName":"Louis","LastName":"Croivébaton","Data":"","YearOfBirth":1754,"YearOfDeath":1793}}
$0000000000001731  -
:  Multi-threaded applications
You can define several @*log@ files per process, and even a per-thread log file, if needed (it could be sometimes handy, for instance on a server running the same logic in parallel in several threads).
The logging settings are made at the logging class level. Each logging class (inheriting from {\f1\fs20 TSynLog}) has its own {\f1\fs20 TSynLogFamily} instance, which is to be used to customize the logging class level. Then you can have several instances of the individual {\f1\fs20 TSynLog} classes, each class sharing the settings of the {\f1\fs20 TSynLogFamily}.
You can therefore initialize the "family" settings before using logging, like in this code which will force to log all levels ({\f1\fs20 LOG_VERBOSE}), and create a per-thread log file, and write the {\f1\fs20 .log} content not in the {\f1\fs20 .exe} folder, but in a custom directory:
! with TSynLogDB.Family do
! begin
!   Level := LOG_VERBOSE;
!!   PerThreadLog := ptOneFilePerThread;
!   DestinationPath := 'C:\Logs';
! end;
If you specifies {\f1\fs20 PerThreadLog := ptIdentifiedInOnFile} for the family, a new column will be added for each log row, with the corresponding {\f1\fs20 ThreadID} - the supplied {\f1\fs20 LogView} tool will handle it as expected. This can be very useful for a multi-threaded server process, e.g. as implement with {\i mORMot}'s Client-Server classes @35@.
:  Log to the console
For debugging purposes, it could be very handy to output the logging content to a console window. It enables interactive debugging of a Client-Server process, for instance: you can interact with the Client, then look in real time at the server console window, and inspect which requests are processed, without the need to open the log file.
The {\f1\fs20 EchoToConsole} property enable you to select which events are to be echoed on the console (perhaps you expect only errors to appear, for instance).
!  with TSQLLog.Family do begin
!    Level := LOG_VERBOSE;
!!    EchoToConsole := LOG_VERBOSE; // log all events to the console
!  end;
Depending on the events, colors will be used to write the corresponding information. Errors will be displayed as light red, for instance.
Note that this echoing process slow down the logging process a lot, since it is currently implemented in a blocking mode, and writing to the console under {\i Windows} is much slower than writing to a file. This feature is therefore disabled by default, and not to be enabled on a production server, but only to make interactive debugging easier.
:104  Remote logging
By default, {\f1\fs20 TSynLog} writes its activity to a local file, and/or to the console. The log file can be transmitted later on (once compressed) to support, for further review and debugging.\line But sometimes, it may be handy to see the logging in real-time, on a remote computer.
You can enable such remote monitoring for a given {\f1\fs20 TSynLog} class, by adding the {\f1\fs20 mORMotHTTPClient.pas} unit in your use clause, then calling the following constructor:
! TSQLHttpClient.CreateForRemoteLogging('192.168.1.15',SQLite3Log,'8091','LogService');
This command will let any {\f1\fs20 SQLite3Log} event be sent to a remote server running at {\f1\fs20 http://192.168.1.15:8091/LogService/RemoteLog} - in fact this should be a {\i mORMot} server, but may be any REST server, able to answer to a {\f1\fs20 PUT} command sent to this URI.
A {\f1\fs20 TSQLHttpClient} instance will be created, and will be managed by the {\f1\fs20 SQLite3Log} instance. It will be released when the application will be closed, or when the {\f1\fs20 SQLite3Log.Family.EchoRemoteStop} method will be called.
In practice, our {\i Log View} tool - see @103@ - is able to run as a compatible remote server. Execute the tool, set the expected {\i Server Root} name ('{\f1\fs20 LogService}' by default), and the expected {\i Server Port} (8091 by default), then click on the "{\f1\fs20 Server Launch}" button.\line The {\i Log View} tool will now display in real time all incoming events, search into their content, and allow to save all received events into a regular {\f1\fs20 .log} or {\f1\fs20 .synlz} file, for further archiving and study.\line Note that since the {\i Log View} tool will run a {\f1\fs20 http.sys} based server - see @88@ - you may have to run once the tool with administrator rights, to register the {\i Server Root} / {\i Server Port} combination for binding.
Implementation of this remote logging has been tuned on both client and server side.\line On client side, log events are gathered and sent in a dedicated background thread: if a lot of events are generated, they will be transferred in chunks of several rows, to minimize resource and bandwidth. On server side, incoming events are stored in memory, and indexed on the fly, with a periodic refresh rate of 500 ms: even a very active client logger will just let the {\i Log View} tool be responsive and efficient.\line Thanks to the nature of the {\f1\fs20 http.sys} based server, several {\i Server Root} URI can be accessed in parallel with several {\i Log View} tool instance, on the same HTTP port: it will ease the IT policy of your network, since a single forwarded port will be able to handle several incoming connections.
See the "{\f1\fs20 RemoteLoggingTest.dpr}" sample from "{\f1\fs20 11 - Exception logging}", in conjunction with the {\f1\fs20 LogView.dpr} tool available in the same folder, for a running example of remote logging.
Note that our cross-platform clients - see @86@ - are able to log to a remote server, with the same exact format as used by our {\f1\fs20 TSynLog} class.
:  Log to third-party libraries
Our {\f1\fs20 TSynLog} class was designed to write its information to a file, and optionally to the console or a remote log server (as we just saw). In fact, {\f1\fs20 TSynLog} is extensively used by the {\i mORMot} framework to provide various levels of details on what happens behind the scene: it is great for debugging purposes.
It may be convenient to let {\f1\fs20 TSynLog} work with any third party logging applications such as {\i CodeSite} or {\i SmartInspect}, or any proprietary solution. As a result, {\i mORMot} logs can be mixed with existing application logs.
You can define the {\f1\fs20 TSynLogFamily.EchoCustom} property to specify a simple event to be triggered for each log operation: the application can then decide to log to a third party logger application.
Note that there is also the {\f1\fs20 TSynLogFamily.NoFile} property, which allows to disable completely the built-in file logging mechanism.
For instance, you may write:
!procedure TMyClass.Echo(Sender: TTextWriter; Level: TSynLogInfo; const Text: RawUTF8);
!begin
!  if Level in LOG_STACKTRACE then // filter only errors
!    writeln(Text); // could be any third-party logger
!end;
!
!...
!  with TSQLLog.Family do begin
!    Level := LOG_VERBOSE;
!    // EchoToConsole := LOG_VERBOSE; // log all events to the console
!!    EchoCustom := aMyClass.Echo; // register third-party logger
!!    NoFile := true; // ensure TSynLog won't use the default log file
!  end;
A process similar to {\f1\fs20 TSynLogFile.ProcessOneLine()} could then parse the incoming {\f1\fs20 Text} value, if needed.
:  Automated log archival
Log archives can be created with the following settings:
! with TSynLogDB.Family do
! begin
!  (...)
!  OnArchive := EventArchiveZip;
!  ArchivePath := '\\Remote\WKS2302\Archive\Logs'; // or any path
! end;
The {\f1\fs20 ArchivePath} property can be set to several functions, taking a timeout delay from the {\f1\fs20 ArchiveAfterDays} property value:
- {\f1\fs20 nil} is the default value, and won't do anything: the {\f1\fs20 .log} will remain on disk until they will be deleted by hand;
- {\f1\fs20 EventArchiveDelete} in order to delete deprecated {\f1\fs20 .log} files;
- {\f1\fs20 EventArchiveSynLZ} to compress the {\f1\fs20 .log} file into a proprietary {\i @*SynLZ@} format: resulting file name will be located in {\f1\fs20 ArchivePath\\log\\YYYYMM\\*.log.synlz}, and the command-line {\f1\fs20 UnSynLz.exe} tool (calling {\f1\fs20 FileUnSynLZ} function of {\f1\fs20 SynCommons.pas} unit) can be used to uncompress it in to plain {\f1\fs20 .log} file;
- {\f1\fs20 SynZip.EventArchiveZip} will archive the {\f1\fs20 .log} files in {\f1\fs20 ArchivePath\\log\\YYYYMM.zip} files, grouping every .
{\i SynLZ} files are less compressed, but created much faster than {\f1\fs20 .zip} files. However, {\f1\fs20 .zip} files are more standard, and on a regular application, compression speed won't be an issue for the application.
:  Log files rotation
You can set {\f1\fs20 TSynLogFamily.RotateFileCount} and {\f1\fs20 RotateFileSizeKB} properties, to enable log file rotation:
- If both values are > 0, the log file will have a fixed name, without any time-stamp within;
- {\f1\fs20 RotateFileSizeKB} will define the maximum size of the main uncompressed log file
- {\f1\fs20 RotateFileCount} will define how many files are kept on disk - note that rotated files are compressed using {\i SynLZ}, so compression will be very fast.
Log file rotation is as easy as:
!  with TSQLLog.Family do begin
!    Level := LOG_VERBOSE;
!    RotateFileCount := 5;        // will maintain a set of up to 5 files
!    RotateFileSizeKB := 20*1024; // rotate by 20 MB logs
!  end;
Such a logging definition will create those files on disk, e.g. for the {\f1\fs20 TestSQL3.dpr} regression tests:
- {\f1\fs20 TestSQL3.log} which will be the latest (current) log file, uncompressed;
- {\f1\fs20 TestSQL3.1.synlz} to {\f1\fs20 TestSQL3.4.synlz} will be the 4 latest log files, after compression. Our {\i Log Viewer} tool - see @103@ - is able to uncompress those {\f1\fs20 .synlz} files directly.
Note that as soon as you active file rotation, {\f1\fs20 PerThreadLog = ptOneFilePerThread} and {\f1\fs20 HighResolutionTimestamp} properties will be ignored, since both features expect a single file to exist per {\f1\fs20 TSynLog} class.
As an alternative, or in addition to this {\i by-size} rotation pattern, you could specify a fixed time of the day to perform the rotation.\line For instance, the following will perform automatic rotation of the log files, whatever their size, at {\f1\fs20 23:00} each evening:
!  with TSQLLog.Family do begin
!    Level := LOG_VERBOSE;
!    RotateFileCount := 5;        // will maintain a set of up to 5 files
!    RotateFileDailyAtHour := 23; // rotate at 11:00 PM
!  end;
If the default behavior - which is to compress all rotated files into {\f1\fs20 .synlz} format, and delete the older files - does not fit your needs, you can set a custom event to the {\f1\fs20 TSynLogFamily.OnRotate} property, which will take care of the file rotation process.
:  Integration within tests
Logging is integrated within the unit @*test@ing classes, so that any failure will create an entry in the log with the source line, and stack trace:
$C:\Dev\lib\SQLite3\exe\TestSQL3.exe 0.0.0.0 (2011-04-13)
$Host=Laptop User=MyName CPU=2*0-15-1027 OS=2.3=5.1.2600 Wow64=0 Freq=3579545
$TSynLogTest 1.13 2011-04-13 05:40:25
$
$20110413 05402559 fail  TTestLowLevelCommon(00B31D70) Low level common: TDynArray "dynamic array failure" stack trace 0002FE0B SynCommons.TDynArray.Init (15148) 00036736 SynCommons.Test64K (18206) 0003682F SynCommons.TTestLowLevelCommon._TDynArray (18214) 000E9C94 TestSQL3 (163)
The difference between a test suit without logging ({\f1\fs20 TSynTests}) and a test suit with logging ({\f1\fs20 TSynTestsLogged}) is only this overridden method:
!procedure TSynTestsLogged.Failed(const msg: string; aTest: TSynTestCase);
!begin
!  inherited;
!  with TestCase[fCurrentMethod] do
!    fLogFile.Log(sllFail,'%: % "%"',
!      [Ident,TestName[fCurrentMethodIndex],msg],aTest);
!end;
In order to enable tests logging, you have just to enable it, e.g. with:
! TSynLogTestLog.Family.Level := LOG_VERBOSE;
You can optionally redirect the following global variable at program initialization, to share testing events with the main {\i mORMot} logs:
!  with TSQLLog.Family do begin
!    Level := LOG_VERBOSE;
!!    TSynLogTestLog := TSQLLog; // share the same log file with whole mORMot
!  end;
:103  Log Viewer
Since the log files tend to be huge (for instance, if you set the logging for our unitary tests, the 17,000,000 test cases do create a huge log file of about 550 MB), a log viewer was definitively in need.
The log-viewer application is available as source code in the "{\i Samples}" folder, in the "{\i 11 - Exception logging}" sub-folder.
:   Open log files
You can run it with a specified log file on the command line, or use the "{\i Browse}" button to browse for a file. That is, you can associate this tool with your {\f1\fs20 .log} files, for instance, and you'll open it just by double-clicking on such files.
Note that if the file is not in our {\f1\fs20 TSynLog} format, it will still be opened as plain text. You'll be able to browse its content and search within, but all the nice features of our logging won't be available, of course.
It is worth saying that the viewer was designed to be {\i fast}.\line In fact, it takes no time to open any log file. For instance, a 390 MB log file is opened in less than one second on my laptop. Under Windows Seven, it takes more time to display the "Open file" dialog window than reading and indexing the 390 MB content.\line It uses internally memory mapped files and optimized data structures to access to the data as fast as possible - see {\f1\fs20 TSynLogFile} class.
:   Log browser
The screen is divided into three main spaces:
- On the left side, the panel of commands;
- On the right side, the log events list;
- On the middle, an optional list of method calls, and another list of threads (not shown by default).
The command panel allows to {\i Browse} your disk for a {\f1\fs20 .log} file. This button is a toggle of an optional {\i Drive / Directory / File} panel on the leftmost side of the tool. When a {\f1\fs20 .log / .synlz / .txt} file is selected, its content is immediately displayed. You can specify a directory name as a parameter of the tool (e.g. in a {\f1\fs20 .lnk} desktop link), which will let the viewer be opened in "Browse" mode, starting with the specified folder.
A button gives access to the global {\i Stats} about its content (customer-side hardware and software running configuration, general numbers about the log), and even ask for a source code line number and unit name from a hexadecimal address available in the log, by browsing for the corresponding {\f1\fs20 .map} file (could be handy if you did not deliver the {\f1\fs20 .map} content within your main executable - which you should have to).
Just below the "{\i Browse}" button, there is an edit field available, with a ? button. Enter any text within this edit field, and it will be searched within the log events list. Search is case-insensitive, and was designed to be fast. Clicking on the ? button (or pressing the {\f1\fs20 F3} key) allows to repeat the last search.
In the very same left panel, you can see all existing events, with its own color and an associated check-box. Note that only events really encountered in the {\f1\fs20 .log} file appear in this list, so its content will change between log files. By selecting / un-selecting a check-box, the corresponding events will be instantaneously displayed / or not on the right side list of events. You can right click on the events check-box list to select a predefined set of events.
The right colored event list follows the events appended to the log, by time order. When you click on an event, its full line content is displayed at the bottom on the screen, in a memo.
Having all @*SQL@ / @*NoSQL@ and @*Client-Server@ events traced in the log is definitively a huge benefit for customer support and bug tracking.
:   Customer-side profiler
One distinctive feature of the {\f1\fs20 TSynLog} logging class is that it is able to map methods or functions entering/leaving (using the {\f1\fs20 Enter} method), and trace this into the logs. The corresponding timing is also written within the "{\i Leave}" event, and allows application profiling from the customer side. Most of the time, profiling an application is done during the testing, with a test environment and database. But this is not, and will never reproduce the exact nature of the customer use: for instance, hardware is not the same (network, memory, CPU), nor the software (Operating System version, [anti-]virus installed)... By enabling customer-side method profiling, the log will contain all relevant information. Those events are named "{\i Enter}" / "{\i Leave}" in the command panel check-box list, and written as + and - in the right-sided event list.
The "{\i Methods profiler}" options allow to display the middle optional method calls list. Several sort order are available: by name (alphabetical sort), by occurrence (in running order, i.e. in the same order than in the event log), by time (the full time corresponding to this method, i.e. the time written within the "{\i Leave}" event), and by proper time (i.e. excluding all time spent in the nested methods).
The "{\i Merge method calls}" check-box allows to regroup all identical method calls, according to their name. In fact, most methods are not called once, but multiple time. And this is the accumulated time spent in the method which is the main argument for code profiling.
I'm quite sure that the first time you'll use this profiling feature on a huge existing application, you'll find out some bottlenecks you will have never thought about before.
:   Per-thread inspection
If the {\f1\fs20 TSynLog} family has specified {\f1\fs20 PerThreadLog := ptIdentifiedInOnFile} property, a new column will be added for each log row, with the corresponding {\f1\fs20 ThreadID} of the logged action.
The log-viewer application will identify this column, and show a "{\i Thread}" group below the left-side commands. It will allow to go to the next thread, or toggle the optional {\i Thread view} list. By checking / un-checking any thread of this list, you are able to inspect the execution log for a given process, very easily. A right-click on this thread list will display a pop-up menu, allowing to select all threads or no thread in one command.
:   Server for remote logging
As was stated above, @104@ can use our {\i Log View} tool as server and real-time viewer for any remote client, either using {\f1\fs20 TSynLog}, or any cross-platform client - see @86@.
Using a remote logging is specially useful from mobile applications (written with {\i Delphi} / @*FireMonkey@ or with @*Smart Mobile Studio@ / @*AJAX@). Our viewer tool allows efficient live debugging of such platforms.
:73  Framework log integration
The framework makes an extensive use of the logging features implemented in the {\f1\fs20 SynLog.pas} unit - see @16@.
In its current implementation, the framework is able to log on request:
- Any exceptions triggered during process, via {\f1\fs20 sllException} and {\f1\fs20 sllExceptionOS} levels;
- Client and server @*REST@ful {\f1\fs20 URL} methods via {\f1\fs20 sllClient} and {\f1\fs20 sllServer} levels;
- @*SQL@ executed statements in the {\i @*SQLite3@} engine via the {\f1\fs20 sllSQL} level;
- @*JSON@ results when retrieved from the {\i SQLite3} engine via the {\f1\fs20 sllResult} level;
- Main errors triggered during process via {\f1\fs20 sllError} level;
- @*Security@ User authentication and @*session@ management via {\f1\fs20 sllUserAuth};
- Some additional low-level information via {\f1\fs20 sllDebug}, {\f1\fs20 sllWarning}  and {\f1\fs20 sllInfo} levels.
Those levels are available via the {\f1\fs20 TSQLLog} class, inheriting from {\f1\fs20 TSynLog}, as defined in @!TSQLLog!Lib\SQLite3\mORMot.pas@.
Three main {\f1\fs20 TSynLogClass} global variables are defined in order to use the same logging family for the whole framework. Since {\f1\fs20 mORMot} units are decoupled (e.g. Database or ORM/SOA), several variables have been defined, as such:
- {\f1\fs20 SynDBLog} for all {\i @*SynDB@*} units, i.e. all generic database code;
- {\f1\fs20 SQLite3Log} for all {\i mORMot*} units, i.e. all @*ORM@ related code;
- {\f1\fs20 SynSQLite3Log} for the {\f1\fs20 SynSQLite3} unit, which implements the {\i @*SQLite3@} engine itself.
By default, redirection to the main {\f1\fs20 TSQLLog} class is done if you use some features within {\i mORMot}:
- {\f1\fs20 mORMot.pas} unit will define {\f1\fs20 SQLite3Log} as its own {\f1\fs20 TSQLLog} class;
- {\f1\fs20 mORMotDB.pas} unit initialization will set {\f1\fs20 SynDBLog := TSQLLog};
- {\f1\fs20 moRMotSQLite3.pas} unit initialization will set {\f1\fs20 SynSQLite3Log := TSQLLog}.
You can set your own class type to {\f1\fs20 SynDBLog / SynSQLite3Log} if you expect separated logging.
As a result, if you execute the following statement at the beginning of {\f1\fs20 TestSQL3.dpr}, regression @*test@s will produce some logging, and resulting into more than 740 MB of log file content, if executed:
!  TSynLogTestLog := TSQLLog; // share the same log file with whole mORMot
!  with TSQLLog.Family do begin
!    Level := LOG_VERBOSE;
!    HighResolutionTimestamp := true;
!    PerThreadLog := ptIdentifiedInOnFile;
!  end;
Creating so much log content won't increase the processing time much. On a recent laptop, whole regression tests process will spent only 2 seconds to write the additional logging, which is the bottleneck of the hard disk writing.
If logging is turned off, there is no speed penalty noticeable.
Logging could be very handy for interactive debug of a client application. Since our {\f1\fs20 TSynLog / TSQLLog} class feature an optional ouput to a console, you are able to see in real-time the incoming requests - see for instance how {\f1\fs20 14 - Interface based services\\Project14ServerHttp.pas} sample is initialized:
!begin
!  // define the log level
!  with TSQLLog.Family do begin
!    Level := LOG_VERBOSE;
!!    EchoToConsole := LOG_VERBOSE; // log all events to the console
!  end;
!  // create a Data Model
!  aModel := TSQLModel.Create([],ROOT_NAME);
!  (...)
Of course, this interactive console refresh slows down the process a lot. It is therefore to be defined only for debugging purposes, not on production.
:44Source code
%cartoon08.png
=[License]
\page
: Availability
As a true {\i Open Source} project, all source code of the framework is available, and latest version can be retrieved from our online repository at @https://synopse.info/fossil
As an alternative, you can monitor or fork our projects from our @*GitHub@ repository, at @http://github.com/synopse/mORMot
The source has been commented following the scheme used by our {\i @*SynProject@} documentation tool. That is all interface definition of the units have special comments, which were extracted then incorporated into this @SAD@, in the following pages.
:  Obtaining the Source Code
Each official release of the framework is available in a dedicated {\f1\fs20 SynopseSQLite3.zip} archive from the official @https://synopse.info web site, but you may want to use the latest version available.
The easiest is to download a nightly-generated archive of the latest version of the trunk, from @https://synopse.info/files/mORMotNightlyBuild.zip
As an alternative, you can manually obtain a {\f1\fs20 .zip} archive containing a snapshot of the latest version of the whole source code tree directly from this repository.
Follow these steps:
- Pointer your web browser at @https://synopse.info/fossil
- Click on the "{\i Login}" menu button.
- Log in as anonymous. The password is shown on screen. Just click on the "{\i Fill out captcha}" button then on the "{\i Login}" button. The reason for requiring this login is to prevent spiders from walking the entire website, downloading ZIP archives of every historical version, and thereby soaking up all our bandwidth.
- Click on the {\i Timeline} or {\i Leaves} link at the top of the page. Preferred way is {\i Leaves} which will give you the latest available version.
- Select a version of the source code you want to download: a version is identified by an hexadecimal link (e.g. {\f1\fs20 6b684fb2}). Note that you must successfully log in as "{\i anonymous}" in steps 1-3 above in order to see the link to the detailed version information.
- Finally, click on the "{\i Zip Archive}" link, available at the end of the "{\i Overview}" header, right ahead to the "{\i Other Links}" title. This link will build a {\f1\fs20 .zip} archive of the complete source code and download it to your browser.
:  Expected compilation platform
The framework source code tree will compile and is tested for the following platforms:
- {\i Delphi} 6 up to the latest {\i Delphi} compiler and IDE version, with {\i @*FreePascal@ Compiler} (FPC) 3.x and {\i @*Lazarus@} support;
- Server side on Windows 32-bit and @**64-bit@ platforms (FPC or {\i Delphi} XE2 and up expected when targeting {\i Win64});
- {\i @*Linux@} 32-bit and 64-bit platform for servers using the FPC 3.2 fixes branch - now stable and tested in production since years (especially {\i @*Debian@/@*Ubuntu@} on {\f1\fs20 x86_64});
- VCL client on Win32/Win64 - GUI may be compiled optionally with third-party non Open-Source @*TMS@ Components, instead of default VCL components - see @http://www.tmssoftware.com/site/tmspack.asp
- @69@ clients on any supported platforms;
- @90@ startup with 2.1, for creating @*AJAX@ / @*JavaScript@ / HTML5 / Mobile clients.
Some part of the library (e.g. {\f1\fs20 SynCommons.pas}, {\f1\fs20 SynTests.pas}, {\f1\fs20 SynLog.pas} {\f1\fs20 SynPDF.pas} or the @27@ units) are also compatible with {\i Delphi} 5.
If you want to compile {\i mORMot} unit into @**packages@, to avoid an obfuscated {\i [DCC Error] @*E2201@ Need imported data reference ($G) to access 'VarCopyProc'} error at compilation, you should defined the {\f1\fs20 USEPACKAGES} conditional in your project's options. Open {\f1\fs20 SynCommons.inc} for a description of this conditional, and all over definitions global to all {\i mORMot} units - see @45@. To avoid related {\i @*E1025@ Unsupported language feature: 'Object'} compilation error, you should probably also set "{\i Generate DCUs only}" in project's options "{\i C/C++ output file generator}".
The framework source code implementation and design tried to be as cross-platform and cross-compiler as possible, since the beginning. It is a lot of work to maintain compatibility towards so many tools and platforms, but we think it is always worth it - especially if you try not depend on {\i Delphi} only, which as shown some backward compatibility issues during its lifetime.
For HTML5 and Mobile clients, our main platform is {\i Smart Mobile Studio}, which is a great combination of ease of use, a powerful {\i SmartPascal} dialect, small applications (much smaller than FMX), with potential packaging as native iOS or {\i Android} applications (via {\i @*PhoneGap@}).
The latest versions of the {\i FreePascal Compiler} together with its great {\i Lazarus} IDE, are now very stable and easy to work with. We don't support {\i CodeTyphon}, since we found some licensing issue with some part of it (e.g. {\i Orca} GUI library origin is doubtful). So we recommend using {\i @*fpcupdeluxe@} - see @203@ - which is maintained by Alfred, a {\i mORMot} contributor. This is amazing to build the whole set of compilers and IDE, with a lot of components, for several platforms (this is a cross-platform project), just from the sources. I like {\i Lazarus} stability and speed much more than {\i Delphi} (did you ever tried to browse and debug {\i included} {\f1\fs20 $I ...} files in the {\i Delphi} IDE? with Lazarus, it is painless), even if the compiler is slower than {\i Delphi}'s, and if the debugger is less integrated and even more unstable than {\i Delphi}'s under Windows (yes, it is possible!). At least, it works, and the {\i Lazarus} IDE is small and efficient. Official {\i @*Linux@} support is available for {\i mORMot} servers, with full features in the {\i FPC} 3.2 branch - we use it on production with {\i Linux} 64-bit since years.
:  SQLite3 static linking for Delphi and FPC
{\i Preliminary note}: if you retrieved the source code from @https://github.com/synopse/mORMot you will have all the needed {\f1\fs20 .obj/.o} static files available in the expected folders. Just ignore this chapter.
In order to maintain our @https://synopse.info/fossil/timeline source code repository in a decent size, we excluded the {\f1\fs20 sqlite3.obj/.o} storage in it, but provide the full source code of the {\i @*SQlite3@} engine in a custom {\f1\fs20 sqlite3.c} file, ready to be compiled with all conditional defined as expected by {\f1\fs20 SynSQlite3Static.pas}. You need to add the official {\i SQlite3} amalgamation file from @https://www.sqlite.org/download.html and put its content into a {\f1\fs20 SQLite3\\amalgamation} sub-folder, for proper compilation. Our custom {\f1\fs20 sqlite3.c} file will add encryption feature to the engine. Also look into {\f1\fs20 SynSQlite3Static.pas} comments if there is any manual patch needed for proper compilation of the amalgamation sourece.
Of course, you are not required to do the compilation: {\f1\fs20 sqlite3.obj} (for {\i Delphi Win32}) and {\f1\fs20 sqlite3.o} files (for {\i Delphi Win64}) are available for {\i Delphi}, as a separated download, from @https://synopse.info/files/sqlite3obj.7z
For Delphi, please download the latest compiled version of these {\f1\fs20 .obj/.o} files from this link. You can also use the supplied {\f1\fs20 c.bat} and {\f1\fs20 c64.bat} files to compile from the original {\f1\fs20 sqlite3.c} file available in the repository, if you have the {\f1\fs20 bcc32}/{\f1\fs20 bcc64} C command-line compiler(s) installed.
For {\i Win32}, the free version works and was used to create the {\f1\fs20 .obj} file, i.e. {\i C++Builder Compiler (bcc compiler) free download} - as available from {\i Embarcadero} web site.
For native {\i Windows} @*64-bit@ applications (since {\i Delphi} XE2), a {\f1\fs20 sqlilte3.o} static file is also available from the same archive. If you need an external dynamic {\f1\fs20 .dll} for Win64, since there is no official {\i SQLite3} download for {\i Win64} yet, you can use the one we supply at @https://synopse.info/files/SQLite3-64.7z
For FPC, you need to download static {\f1\fs20 .o} files from @https://synopse.info/files/sqlite3fpc.7z then uncompress the embedded {\f1\fs20 static} folder and its sub-folders at the {\i mORMot} root folder (i.e. where {\f1\fs20 Synopse.inc} and {\f1\fs20 SynCommons.pas} stay). If you retrieved the source code from our @*GitHub@ repository at @https://github.com/synopse/mORMot you already got the {\f1\fs20 static} sub-folder as expected by the framework. Those {\f1\fs20 static} files have been patched to support optional encryption of the {\i SQLite3} database file. Then enable the {\f1\fs20 FPCSQLITE3STATIC} conditional in your project, or directly modify {\f1\fs20 Synopse.inc} to include it, so that those {\f1\fs20 .o} files will be statically linked to the executable.
You could also compile the static libraries from the {\f1\fs20 sqlite3.c} source, to run with FPC - do not forget to enable the {\f1\fs20 FPCSQLITE3STATIC} conditional in this case also.\line Under {\i Windows}, ensure the {\i MinGW} compiler is installed, then execute {\f1\fs20 c-fpcmingw.bat} from the {\i SQLite3} folder. It will create the {\f1\fs20 sqlite3.o} and {\f1\fs20 sqlite3fts.o} files, as expected by FPC.\line Under {\i @*Linux@}, Use the {\f1\fs20 c-fpcgcclin.sh} bash script.
:  SpiderMonkey library
To enable {\i @*JavaScript@} support in {\i mORmot}, we rely on our version of the {\i @*SpiderMonkey@} library. See @79@.
You can download the needed files from @https://synopse.info/files/synsm.7z
Do not forget to copy both files in the executable folder. For instance, put both {\f1\fs20 mozjs.dll} and {\f1\fs20 nspr4.dll} files with your {\f1\fs20 JSHttpApiServer.exe}.
By now, this library will work only under {\i Win32}, with {\i Delphi} as compiler - it has not yet been tested nor ported to FPC and other platforms.
:  Folder layout
As retrieved from our source code repository, you'll find the following folder layout:
\graph mORMotSourceCodeFolders mORMot Source Code Folders
\root\SQlite3
\SQlite3\Samples
\SQlite3\Documentation
\root\SynDBDataset
\root\CrossPlatform
\
In fact, you will get:
|%30%70
|\b Directory|Description\b0
|{\f1\fs20 /}|Root folder, containing common files
|{\f1\fs20 CrossPlatform}|Contains code for cross-platform clients
|{\f1\fs20 HtmlView/}|A fork of the freeware {\f1\fs20 THtmlView} component, used as a demo of the {\f1\fs20 SynPdf} unit - not finished, and not truly Unicode ready
|{\f1\fs20 LVCL/}|{\i Light VCL} replacement files for standard VCL (for {\i Delphi} 6-7 only)
|{\f1\fs20 RTL7/}|Enhanced RTL .dcu for {\i Delphi} 7 (not mandatory at all), and {\i @*FastMM4@} memory manager to be used before {\i Delphi} 2006
|{\f1\fs20 SQLite3/}|Contains all @*ORM@ / @*SOA@ related files of the framework (i.e. {\i mORMot} itself) and its documentation
|{\f1\fs20 SynDBDataset/}|{\f1\fs20 DB.pas}-based external database providers
|{\f1\fs20 SynProject/}|Source code of the {\i @*SynProject@} tool, used to edit and generate this documentation
;|{\f1\fs20 zeos/}|A fork of the freeware {\i Zeos} library - not finished, and not truly Unicode ready
|%
:   Root folder
In the {\i Root folder}, some common files are defined:
|%30%70
|\b File|Description\b0
|{\f1\fs20 CPort.*}|A fork of the freeware {\i ComPort} Library ver. 2.63
|{\f1\fs20 PasZip.pas}|ZIP/LZ77 Deflate/Inflate Compression in pure pascal ({\f1\fs20 SynZip.pas} is faster)
|{\f1\fs20 SynBigTable.pas}|class used to store huge amount of data with fast retrieval
|{\f1\fs20 SynBz.pas bunzipasm.inc}|fast BZ2 compression/decompression
|{\f1\fs20 SynBzPas.pas}|pascal implementation of BZ2 decompression
|{\f1\fs20 SynCommons.pas}|common functions used by most Synopse projects
|{\f1\fs20 SynCrtSock.pas SynBidirSock.pas}|classes implementing @*HTTP@ and @*WebSockets@ client and server protocol
|{\f1\fs20 SynCrypto.pas}|fast cryptographic routines (hashing and cypher)
|{\f1\fs20 SynDprUses.inc}|generic header included in the beginning of the uses clause of a .dpr source code
|{\f1\fs20 SynEcc.pas}|certificate-based public-key cryptography using ECC-secp256r1
|{\f1\fs20 SynGdiPlus.pas}|GDI+ library API access with anti-aliasing drawing
|{\f1\fs20 SynLog.pas}|logging functions used by most Synopse projects
|{\f1\fs20 SynLizard.pas}|Lizard (LZ5) compression/decompression unit
|{\f1\fs20 SynLZ.pas}|@**SynLZ@ compression/decompression unit - used by {\f1\fs20 SynCommons.pas}
|{\f1\fs20 SynLZO.pas}|LZO compression/decompression unit
|{\f1\fs20 SynMemoEx.pas}|Synopse extended {\f1\fs20 TMemo} visual component (used e.g. in {\i @*SynProject@}) - for pre-Unicode {\i Delphi} only
|{\f1\fs20 SynMongoDB.pas}|Direct {\i @*MongoDB@} @*NoSQL@ database access
|{\f1\fs20 SynMustache.pas}|{\i @*Mustache@} logic-less template engine
|{\f1\fs20 SynPdf.pas}|@*PDF@ file generation unit
|{\f1\fs20 SynScaleMM.pas}|multi-thread friendly memory manager unit - not finished yet
|{\f1\fs20 SynSelfTests.pas}|automated @*test@s for {\i mORMot} Framework
|{\f1\fs20 SynSMAPI.pas}|{\i SpiderMonkey} JavaScript engine API definition
|{\f1\fs20 SynSM.pas}|{\i SpiderMonkey} JavaScript engine higher level classes
|{\f1\fs20 SynSQLite3.pas}|{\i @*SQLite3@} embedded Database engine
|{\f1\fs20 SynSQLite3Static.pas}|statically linked @*SQLite3@ engine (with associated {\f1\fs20 .obj/.o})
|{\f1\fs20 SynSSPIAuth.pas}|low level access to Windows Authentication
|{\f1\fs20 SynTaskDialog.*}|implement TaskDialog window\line (native on Vista/Seven, emulated on XP)
|{\f1\fs20 SynTests.pas}|cross-compiler unitary tests functions
|{\f1\fs20 SynWinSock.pas}|low level access to network Sockets for the Windows platform
|{\f1\fs20 SynZip.pas deflate.obj trees.obj}|low-level access to ZLib compression, 1.2.5
|{\f1\fs20 SynZipFiles.pas}|high-level access to .zip archive file compression
|{\f1\fs20 Synopse.inc}|generic header to be included in all units to set some global conditional definitions
|{\f1\fs20 vista.*}|A resource file enabling theming under XP
|{\f1\fs20 vistaAdm.*}|A resource file enabling theming under XP and Administrator rights under Vista
|%
In the same {\i Root folder}, the external database-agnostic units are located:
|%30%70
|\b File|Description\b0
|{\f1\fs20 @*SynDB@.pas}|abstract database direct access classes
|{\f1\fs20 SynOleDB.pas}|fast @*OleDB@ direct access classes
|{\f1\fs20 SynDBODBC.pas}|fast @*ODBC@ direct access classes
|{\f1\fs20 SynDBOracle.pas}|{\i @*Oracle@} DB direct access classes (via OCI)
|{\f1\fs20 SynDBSQLite3.pas}|{\i @*SQLite3@} direct access classes
|{\f1\fs20 SynDBDataset.pas}|{\f1\fs20 @*TDataSet@} / {\f1\fs20 @*TQuery@}-like access classes\line (drivers included in {\f1\fs20 SynDBDataset} sub-folder)
|{\f1\fs20 SynDBRemote.pas}|remote access over HTTP
|{\f1\fs20 SynDBVCL.pas}|DB VCL read-only dataset using {\f1\fs20 SynDB.pas} data access
|{\f1\fs20 SynDBZEOS.pas}|{\i @*Zeos@Lib} / ZDBC direct access classes
|%
:   SynDBDataset folder
In a {\f1\fs20 SynDBDataset} folder, some external database providers are available, to be used with the {\f1\fs20 SynDBDataset.pas} classes:
|%30%70
|\b File|Description\b0
|{\f1\fs20 SynDBBDE.pas}|{\i BDE} access classes
|{\f1\fs20 SynDBNexusDB.pas}|{\i @*NexusDB@} access classes
|{\f1\fs20 SynDBFireDAC.pas}|{\i @*FireDAC@} / {\i @*AnyDAC@} library access classes
|{\f1\fs20 SynDBUniDAC.pas}|{\i @*UniDAC@} library access classes
|%
:   SQLite3 folder
In the {\f1\fs20 SQlite3/} folder, the files implementing the {\i Synopse mORMot framework} itself, i.e. its @*ORM@ and @*SOA@ features (using units from the {\i Root folder}):
|%30%70
|\b File|Description\b0
|{\f1\fs20 Documentation/}|Sub folder containing the source of the framework documentation
|{\f1\fs20 Samples/}|Sub folders containing some sample code
|{\f1\fs20 mORMot.pas}|Main @*ORM@ / @*SOA@ unit of the framework
|{\f1\fs20 mORMotDB.pas}|Virtual Tables for ORM external {\f1\fs20 SynDB.pas} access
|{\f1\fs20 mORMotFastCgiServer.pas}|FastCGI server - not fully tested
|{\f1\fs20 mORMotHttpClient.pas}|HTTP/1.1 Client
|{\f1\fs20 mORMotHttpServer.pas}|HTTP/1.1 Server
|{\f1\fs20 mORMotReport.pas}|Integrated Reporting engine
|{\f1\fs20 mORMotService.pas}|Stand-alone Service
|{\f1\fs20 mORMotSQLite3.pas}|{\i SQLite3} kernel bridge between {\f1\fs20 mORMot.pas} and {\f1\fs20 SynSQLite3.pas}
|{\f1\fs20 mORMoti18n.pas}|internationalization (@*i18n@) routines and classes
|{\f1\fs20 mORMotMongoDB.pas}|@*ODM@ integration of {\i @*MongoDB@} @*NoSQL@ database
|{\f1\fs20 mORMotMVC.pas}|@*MVC@ classes to develop high performance @*Web Application@s
|{\f1\fs20 mORMotToolBar.pas}|ORM ToolBar User Interface generation
|{\f1\fs20 mORMotUI.*}|Grid to display Database content
|{\f1\fs20 mORMotUIEdit.*}|Record edition dialog, used to edit record content on the screen
|{\f1\fs20 mORMotUILogin.*}|some common User Interface functions and dialogs
|{\f1\fs20 mORMotUIOptions.*}|General Options setting dialog, generated from code
|{\f1\fs20 mORMotUIQuery.*}|Form handling queries to a User Interface Grid, using our ORM RTTI to define search parameters and algorithms
|{\f1\fs20 mORMotVCL.pas}|DB VCL dataset using {\f1\fs20 TSQLTable/TSQLTableJSON} data access
|{\f1\fs20 *.bmp *.rc}|Resource files, compiled into {\f1\fs20 *.res} files
|{\f1\fs20 TestSQL3.dpr\line mORMotSelfTests.pas}|Main @*test@ing program of the Synopse {\i mORMot} framework
|{\f1\fs20 TestSQL3Register.dpr}|Run as administrator for {\i TestSQL3} to use {\i @*http.sys@} on Vista/Seven
|{\f1\fs20 c.bat sqlite3.c}|Source code of the {\i SQLite3} embedded Database engine
|%
:   CrossPlatform folder
In a {\f1\fs20 CrossPlatform} folder, some source code is available, to be used when creating {\i mORMot} clients for compilers or platforms not supported by the main branch:
|%40%60
|\b File|Description\b0
|{\f1\fs20 SynCrossPlatform.inc}|Includes cross-platform and cross-compiler conditionals
|{\f1\fs20 SynCrossPlatformJSON.pas}|Cross-platform @*JSON@ support for {\i Delphi} and FPC
|{\f1\fs20 SynCrossPlatformREST.pas}|Main unit, handling secured ORM and SOA RESTful client
|{\f1\fs20 SynCrossPlatformCrypto.pas}|@*SHA256@ and @*crc32@ algorithms, used for authentication
|{\f1\fs20 SynCrossPlatformSpecific.pas}|System-specific functions, e.g. HTTP clients
|%
See @86@ for more information.
\page
:113 Delphi Installation
{\i Note: for FPC setup, see @125@.}
To setup mORMot for {\i Delphi 6} up to the latest {\i Delphi} version, you have two ways: either download the framework from archives, or clone our {\i GitHub} repository at @https://github.com/synopse/mORMot
:  Manual download
Download and uncompress the framework archives, including all sub-folders, into a local directory of your computer (for instance, {\f1\fs20 D:\\Dev\\mORMot}).
|%70
|{\b Snapshot of the latest source code repository}\line\tab @https://synopse.info/files/mORMotNightlyBuild.zip \line\tab into {\f1\fs20 D:\\Dev\\mORMot\\} (including all sub-folders)
|{\b for Delphi: static SQLite3 {\f1\fs20 .obj/.o} files}\line\tab @https://synopse.info/files/sqlite3obj.7z \line\tab into {\f1\fs20 D:\\Dev\\mORMot\\SQLite3\\}
|{\b for FPC: static {\f1\fs20 .o} files for Windows or Linux/BSD}\line\tab @https://synopse.info/files/sqlite3fpc.7z \line\tab whole {\f1\fs20 static} folder into {\f1\fs20 D:\\Dev\\mORMot\\}
|{\b optional 64-bit SQlite3 external library}\line\tab @https://synopse.info/files/SQLite3-64.7z \line\tab into your Win64 {\f1\fs20 .exe} folders
|{\b 32-bit SpiderMonkey library}\line\tab @https://synopse.info/files/synsm.7z \line\tab into your {\f1\fs20 .exe} folders needing JavaScript
|%
:  Get from GitHub
Or you may just clone our {\i @**GitHub@} repository, from @https://github.com/synopse/mORMot e.g. via:
$ d:
$ cd Dev
$ git clone https://github.com/synopse/mORMot.git
It will create a {\f1\fs20 d:\\Dev\\mORMot} local folder, which will eventually be re-synchronized with the official sources. Advantage of cloning our {\i GitHub} repository is that it contains binaries for static linking, ({\i SQLite3} and @*FPC@ specific), in a single step.
Just take care that if you downloaded some other library from Synopse (e.g. from @https://github.com/synopse/SynPDF or @https://github.com/synopse/dmustache), you should better use the main @https://github.com/synopse/mORMot only, which contains other projects, to avoid any version confusion. We have seen a lot of installation problems reported in our forum due to source code file collision from several repositories, not in the same revision.
:  Setup the Delphi IDE
To let your IDE know about {\i mORMot} source code, add the following paths to your {\i Delphi} IDE (in {\i Tools/Environment/Library} or {\i Tools/Options/Language/Delphi Options/Library} menu depending on your Delphi version):
- {\i Library path}:\line{\i (...existing path...)\f1\fs20 ;D:\\Dev\\mORMot;D:\\Dev\\mORMot\\SQLite3;D:\\Dev\\mORMot\\SynDBDataset}
- {\i Search path}:\line{\i (...existing path...)\f1\fs20 ;D:\\Dev\\mORMot;D:\\Dev\\mORMot\\SQLite3;D:\\Dev\\mORMot\\SynDBDataset}
For any cross-platform client, do not forget to include the {\f1\fs20 D:\\Dev\\mORMot\\CrossPlatform} to the {\i Delphi} or {\i FreePascal} IDE paths of the corresponding targets.\line For {\i @*Smart Mobile Studio@}, execute {\f1\fs20 CopySynCrossPlatformUnits.bat} to set the needed units in the IDE repository.
Note that before {\i Delphi} 2006, you will need to download and install {\i @*FastMM4@} heap memory manager - from @http://sourceforge.net/projects/fastmm or from the {\f1\fs20 D:\\Dev\\mORMot\\RTL7} sub folder of our repository - for some samples to work (without it, {\i mORMot} units will work, but will be slower). Starting with {\i Delphi} 2006, {\i FastMM4} is already included within the system RTL, so you do not need to download it.
Open the {\f1\fs20 TestSQL3.dpr} program from the {\f1\fs20 SQLite3} sub-folder. You should be able to compile it and run all regression @*test@s on your computer.\line If you want to run the tests with the fast @*http.sys@ kernel-based HTTP server, you'll need to compile and run (as administrator) {\f1\fs20 TestSQL3Register.dpr} once before launching {\f1\fs20 TestSQL3.dpr}.
Then open the {\f1\fs20 *.dpr} files, as available in the {\f1\fs20 SQLite3\\Samples} sub-folder. You should be able to compile all sample programs, including {\f1\fs20 SynFile.dpr} in the {\f1\fs20 MainDemo} folder.
Enjoy!
\page
:125 FreePascal / Lazarus Installation
{\i Note: see also @113@.}
:202  Possible targets
You can use the {\i @**FreePascal@ Compiler} (@**FPC@) to (cross-)compile the {\i mORMot} framework source code, targeting the following CPU and OS combinations:
- i386-win32
- x86_64-win64
- i386-linux
- x86_64-linux
- i386-freebsd
- x86_64-freebsd
- i386-darwin
- x86_64-darwin
- arm-linux
- aarch64-linux
32-bit and 64-bit Windows and Linux platforms are the main supported targets, used in production since years. Others may need some enhancements, and you are free to contribute! {\i mORMot} has been reported to work on a Raspberry Pi running Linux, thanks to FPC abilities - and with good performance and stability.
{\i Linux} is a premium target for cheap and efficient server @75@. Since {\i mORMot} has no dependency, installing a new {\i mORMot} server is as easy as copying its executable on a blank {\i Linux} host, then run it. No need to install any framework nor runtime. Even the {\i SQLite3} engine will be statically linked on most platforms, as we provide up-to-date binaries in our repository. You could even use diverse operating systems (several {\i Linux} or {\i Windows Server} versions) in your {\i mORMot} servers farm, with minimal system requirements, and updates.
For proper FPC compilation, ensure you have the following settings to your project:
- {\i Other unit files (-Fu)}:\line{\i \f1\fs20 D:\\Dev\\mORMot;D:\\Dev\\mORMot\\SQLite3;D:\\Dev\\mORMot\\SQLite3\\DDD\\infra}
- {\i Include files (-Fi)}:\line{\i \f1\fs20 $(ProjOutDir);D:\\Dev\\mORMot;D:\\Dev\\mORMot\\SQLite3}
- {\i Libraries (-fFl)}:\line{\i \f1\fs20 D:\\Dev\\mORMot\\static\\$(TargetCPU)-$(TargetOS)}
Replace {\f1\fs20 D:\\Dev\\mORMot} path by the absolute/relative folder where you did install the framework. In practice, a relative path (e.g. {\f1\fs20 ..\\..\\mORMot}) is preferred.
:203  Setup your dedicated FPC / Lazarus environment with fpcupdeluxe
We currently use the FPC 3.2 fixes branch compiler, and the corresponding {\i Lazarus} IDE.
If you want to use @80@, ensure that your revision includes the fix for @http://mantis.freepascal.org/view.php?id=26773 bug, i.e. newer than revision 28995 from 2014-11-05T22:17:54. This bug was not fixed in 2.6.4 branch, but any newer 3.x revision should be enough.
But since the FPC trunk may be unstable, we will propose to put in place a stable development environment based on the FPC 3.2 branch to work with your {\i mORMot}-based projects. It may ease support and debugging.
For this task, don't download an existing binary release of FPC / Lazarus, but use the {\i @**fpcupdeluxe@} tool, as published at @http://wiki.freepascal.org/fpcupdeluxe - it will allow to build your environment directly from the sources, and install it in a dedicated folder. Several FPC / Lazarus installations, with dedicated revision numbers, may coexist on the same computer: just ensure you run Lazarus from the shortcut created by {\i fpcupdeluxe}.
- Download the latest release of the tool from @https://github.com/LongDirtyAnimAlf/fpcupdeluxe/releases
- Unpack it in a dedicated folder, and run its executable.
- On the main screen, locate on the left the two versions listboxes. Select "{\f1\fs20 3.2}" for {\i FPC version} and "{\f1\fs20 2.1.0}" for {\i Lazarus version}.
- Important note: if you want to cross-compile from Windows to other systems, e.g. install a Linux cross-compiler on Windows, ensure you installed the {\i Win32} FPC compiler and Lazarus, {\i not the Win64} version, which is known to have troubles with {\f1\fs20 currency} support;
- Then build the FPC and Lazarus binaries directly from the latest sources, by clicking on "Install/update FPC+Laz".
Those branches are currently used for building our production projects, so are expected to be properly tested and supported. \line At the time of the writing of this documentation, our Lazarus IDE (on Linux) reports using:
- FPC SVN 45643 (3.2.0)
- Lazarus SVN 64940 (2.1.0).
One big advantage of {\i fpcupdeluxe} is that you can very easily install cross-compilers for the CPU / OS combinations enumerated at @202@.\line Just go to the "Cross" tab, then select the target systems, and click on "Install compiler".\line It may be needed to download the cross-compiler binaries (once): just select "Yes" when prompted.
You could install {\i mORMot} using {\i fpcupdeluxe}, but we recommend you clone our @https://github.com/synopse/mORMot repository, and setup the expected project paths, as detailed above at @113@.
If you don't want to define a given version, the current {\i trunk} should/could work, if it didn't include any regression at the time you get it - this is why we provide "supported" branches.\line If you want to use the {\i FPC trunk}, please modify line #262 in {\f1\fs20 Synopse.inc} to enable the {\f1\fs20 FPC_PROVIDE_ATTR_TABLE} conditional and support the latest trunk RTTI changes:
!  {$if not defined(VER3_0) and not defined(VER3_2) and not defined(VER2)}
!!    {$define FPC_PROVIDE_ATTR_TABLE} // to be defined since SVN 42356-42411
!    // on compilation error in SynFPCTypInfo, undefine the above conditional
!    // see https://lists.freepascal.org/pipermail/fpc-announce/2019-July/000612.html
!  {$ifend}
Sadly, there is no official conditional available to have this RTTI change detected. You need to define globally this conditional.
:  Missing RTTI for interfaces in old FPC 2.6
Sadly, if you use a somewhat old revision of FPC, you may have to face some long-time unresolved FPC compiler-level restriction/issue, which did not supply the needed {\f1\fs20 interface} RTTI, which was available since Delphi 6 - see @http://bugs.freepascal.org/view.php?id=26774 \line As a consequence, SOA, mock/stub and MVC framework features will not work directly with older FPC revisions.
You could upgrade to a more recent FPC - we encourage you to @203@ - or we will propose here a workaround to compile such {\i mORMot} applications with oldest FPC. The trick is to use Delphi to generate one unit containing the needed information.
The {\f1\fs20 mORMotWrappers.pas} unit proposes a {\f1\fs20 ComputeFPCInterfacesUnit()} function, which could be used on Delphi to generate the RTTI unit for FPC, as such:
- Ensure that the application will use all its needed {\f1\fs20 interface}: for instance, run all your regression tests, and/or use all its SOA/MVC features if you are not confident about your test coverage;
- Just before the application exits, add a call to {\f1\fs20 ComputeFPCInterfacesUnit()} with the proper folders, e.g. at the very end of your {\f1\fs20 .dpr} code.
For instance, here is how {\f1\fs20 TestSQL3.dpr} has been modified:
!program TestSQL3;
! ...
!uses
!  ...
!!  mORMotWrappers.pas,
!  ...
!begin
!  SQLite3ConsoleTests;
!  {$ifdef COMPUTEFPCINTERFACES}
!  ChDir(ExtractFilePath(ParamStr(0)));
!!  ComputeFPCInterfacesUnit(
!!    ['..\CrossPlatform\templates','..\..\CrossPlatform\templates'],
!!     '\..\..\SQlite3\TestSQL3FPCInterfaces.pas');
!  {$endif}
!end.
If you define the {\f1\fs20 COMPUTEFPCINTERFACES} conditional, the {\f1\fs20 TestSQL3FPCInterfaces.pas} unit will be generated.
Of course, for your own application, you may use absolute path names: here we used relative naming, via {\f1\fs20 ..\\}, so that it will work on any development folder configuration.
Then, add it to any of your uses clause, as such:
!uses
!  ...
!!  TestSQL3FPCInterfaces, // will register RTTI for interfaces under FPC
!  ...
This unit will do nothing when compiled under {\i Delphi}: it will register the RTTI only when compiled with {\i FPC}.
The rest of your code will be untouched, and could be shared between Delphi and FPC.
If you do not modify the {\f1\fs20 interface} methods definition, this generation step could be safely bypassed.
We hope that in a close future, the FPC team will fix the @http://bugs.freepascal.org/view.php?id=26774 issue, but the ticket seems pretty inactive since its creation.
:143  Writing your project for FPC
If you want your application to compile with FPC, some little patterns should be followed.
In all your source code file, the easiest is to including the following {\f1\fs20 mORMot} file, which will define all compiler options and conditionals as expected:
!{$I Synopse.inc} // define HASINLINE USETYPEINFO CPU32 CPU64 OWNNORMTOUPPER
Then in your {\f1\fs20 .dpr} file, you should write:
!uses
!  {$ifdef FPC} // we may be on Kylix or upcoming Delphi for Linux
!  {$ifdef Linux}
!  // if you use threads
!  cthreads,
!  // widestring manager for Linux if needed !!
!  // could also be put in another unit ... but doc states: as early as possible
!  cwstring, // optional
!  {$endif}
!  {$endif}
In fact, these above lines have been added to {\f1\fs20 @*SynDprUses.inc@}, so you may just write the following:
!uses
!  {$I SynDprUses.inc}    // will enable FastMM4 prior to Delphi 2006, and enable FPC on linux
As a side benefit, you will be able to share the same {\f1\fs20 .dpr} with Delphi, and it will enable {\i @*FastMM4@} for older versions which do not include it as default heap manager.
For instance a minimal FPC project to run the regression tests may be:
!program LinuxSynTestFPCLinuxi386;
!
!{$I Synopse.inc}
!{$APPTYPE CONSOLE}
!
!uses
!  {$I SynDprUses.inc}
!  mORMotSelfTests;
!
!begin
!  SQLite3ConsoleTests;
!end.
In your user code, ensure you do not directly link to the {\f1\fs20 Windows} unit, but rely on the cross-platform classes and functions as defined in {\f1\fs20 SysUtils.pas}, {\f1\fs20 Classes.pas} and {\f1\fs20 SynCommons.pas}. You could find in {\f1\fs20 SynFPCTypInfo.pas} and {\f1\fs20 SynFPCLinux.pas} some low-level functions dedicated to FPC and {\i Linux} compilation, to be used with legacy units - your new code should better rely on higher level functions and classes.
If you rely on {\i mORMot} classes and types, e.g. use {\f1\fs20 RawUTF8} for all your {\f1\fs20 string} process in the business logic, and do not use Delphi-specific features (like generics, or new syntax sugar), it will be very easy to let your application compile with FPC.
;In practice, we use Delphi as our main IDE, then switch to Lazarus under a small integrated {\i Linux VirtualBox}, running a low resource {\i XFCE} desktop. We defined the {\i Windows} folder containing the source as a {\i VirtualBox} shared folder, so that we are able to compile, debug and test the {\i Linux} version of any executable in native {\i Linux}, on the same computer, from the very same sources. We found out that {\i Lazarus} debugging was pretty smooth on {\i Linux} - GDB is smoother on {\i Linux} than {\i Windows}, by the way. Then switching from {\i Delphi/Windows} to {\i Lazarus/Linux} is direct and natural, especially when the {\i VirtualBox} "Integrated Desktop" feature is enabled.
:142  Linux VM installation tips
Here are a few informal notes about getting running a FPC/Lazarus virtual machine running {\i XUbuntu}, on a {\i Windows} host. They are published as a general guideline, and we will not provide any reference procedure, nor support it. As stated in @203@, instead of using a virtual machine, you could just install the needed cross-compilers, then generate your Linux/BSD executables from your Windows Lazarus.
- Install the latest {\i VirtualBox} version from @http://www.virtualbox.org/ to Windows;
- Download the latest {\f1\fs20 .iso} version published at @http://xubuntu.org/ or any other place - we use XFCE since it is a very lightweight desktop, perfect to run {\i Lazarus}, and we selected an Ubuntu LTS revision (14.04 at the time of this writing), which will be the same used on Internet servers;
- Create a new virtual machine (VM) in {\i VirtualBox}, with 1 or 2 CPUs, more than 512 MB of RAM (we use 777 MB), and an automatic-growing disk storage, with a maximal size of 15 GB; ensure that the disk storage is marked as SSD if your real host storage is a SSD;
- Let the CDROM storage point to the {\f1\fs20 .iso} you downloaded;
- Start the VM and install {\i Linux} locally, as usual - you may select to download the updated packages during the installation, for safety;
- When the system restarts, if it asks for software updates, accept and wait for the update installation to finish - it is a good idea to have the latest version of the kernel and libraries before installing the {\i VirtualBox} drivers;
- Restart your VM when asked to;
- Under a {\i @*Ubuntu@/@*Debian@} terminal, write the following commands:
$ sudo apt-get update
$ sudo apt-get upgrade
$ sudo apt-get install dkms
- Restart the VM, then select "Insert Guest Additions CD image" from the VM "Devices" menu: a virtual CD will be mounted on your system and appear on your desktop;
- Run the following command, according to your current user name and {\i VirtualBox} version:
$ sudo sh /media/...user.../VBOXADDITIONS_..../VBoxLinuxAdditions.run
- Restart the VM, then add a permanent shared folder in the VM configuration, named {\f1\fs20 Lib}, and pointing to your local {\i mORMot} installation (e.g. {\f1\fs20 d:\\Dev\\mORMot};
- Create a void folder, e.g. in your home:
$ mkdir lib
- Create a launcher for the following command, to mount the shared folder as expected:
$ sudo mount -t vboxsf lib /home/...user.../lib
- Execute the following commands:
$ sudo apt-get install build-essential mingw32-binutils subversion libgtk2.0-dev
$ sudo ln -s /usr/bin/i586-mingw32msvc-windres /usr/bin/windres
- Then install FPC / Lazarus as detailed in @203@
- If you have issues during SVN retrieval, go the {\f1\fs20 development/fpc} folder, then run the following before trying again the {\f1\fs20 fpcup_linux_x86} command:
$ svn cleanup
$ svn update
If you followed the above steps, you should now have the expected Lazarus IDE and the corresponding FPC compiler. It is amazing seeing the whole compiler + IDE being compiled from the official sources, for free, and in a few minutes.
: CrossKylix support
:  What is Cross-Kylix?
The framework source code can also be cross-compiled under Delphi into a {\i @*Linux@} executable, using {\i @**CrossKylix@}.\line @https://crosskylix.untergrund.net is a free toolkit to integrate the Borland {\i Kylix} ({\i Delphi} for {\i Linux}) compiler into the Delphi Windows IDE.
{\i CrossKylix} has indeed several known drawbacks:
- It is a dead project, but an alive product. It still works!
- You can not buy it any more. {\i Kylix} 3 was shipped with Delphi 7.
- You need an actual {\i Kylix} CD (or an ISO image) to install it, since {\i CrossKylix} is just a wrapper around the official compiler, to let it run under Windows.
- Visual applications (based on the CLX framework - the predecessor of FMX) may still compile, but should not be used. But for server applications, it is still a pretty viable solution.
- The debugger and IDE is unusable. But thanks to our {\f1\fs20 SynLog.pas}, you can debug your applications, with a full stack trace in the log, in case of any exception.
We added {\i CrossKylix} support for several reasons:
- We use it since years, with great success, so we know it better than FPC.
- It has still a better compiler than FPC, e.g. for the RTTI we need on interfaces, or even for executable size and memory use.
- Its compilation is instant - whereas FPC is long to compile.
- It supports {\f1\fs20 FastMM4}, which performs better than the FPC memory manager, from our tests.
- Resulting executables, for {\i mORMot} purpose, are faster than FPC - timing based on the regression tests.
- If the code works with Delphi 7, it will certainly work with {\i Kylix} (since it shares the same compiler and RTL), whereas FPC is compatible, but not the same. In particular, it does not suffer from limited RTTI or other FPC limitations. So it sounds safer to be used on production than FPC, even today.
- There is not a lot of {\f1\fs20 IFDEF}, but in {\f1\fs20 SynCommons.pas}. Then there is a {\f1\fs20 SynKylix.pas} unit for several functions. User code will be the same than Delphi and FPC.
- There is a {\i Linux} compiler just released by {\i Embarcadero} since latest Delphi, but an {\i Entreprise} license is required, so we currently skip its support, and focus on FPC...
Currently, we use FPC with success for building {\f1\fs20 i386} and {\f1\fs20 x86_64} executables. FPC is therefore recommended for production work targeting {\i Linux}. See @125@ and @203@.
Once you have installed {\i CrossKylix}, and set up its search path to the same as Delphi - see @113@, you should be able to compile your project for {\i Linux}, directly from your {\i Delphi} IDE. Then you need an actual {\i Linux} system to test it - please check the @142@.
A minimal console application which will compile for both {\i Delphi} and {\i CrossKylix}, running all our regression tests, may be:
!program Test;
!
!{$APPTYPE CONSOLE}
!
!uses
!  FastMM4, // optional - only for CrossKylix or Delphi < 2006
!  mORMotSelfTests;
!
!begin
!  SQLite3ConsoleTests;
!end.
Similar guidelines as for @143@ do apply with {\i CrossKylix}. In particular, you should never use the {\f1\fs20 Windows} unit in your server code, but rely on the cross-platform classes and functions as defined in {\f1\fs20 SysUtils.pas}, {\f1\fs20 Classes.pas} and {\f1\fs20 SynCommons.pas}.
We did not succeed to have a static {\i SQLite3} library linked by the {\i Kylix} compiler. It compiles about the {\f1\fs20 .o} format - sounds like if its linker expects a {\f1\fs20 gcc2} format (which is nowadays deprecated), and does not accept the {\f1\fs20 gcc3} or {\f1\fs20 gcc4} generated binaries. So you need to install the {\i sqlite3} as external library on your {\i Linux}.
On a 32-bit system, it is just a one line - depending on your distribution, here {\i Ubuntu}:
$ sudo apt-get install sqlite3
For a 64-bit system, you need to explicitly install the x86 32-bit version of {\i SQlite3}:
$ sudo apt-get install sqlite3:i386
or download and install manually packages for both modes:
$ sudo dpkg -i libsqlite3-0_3.8.2-1ubuntu2_amd64.deb libsqlite3-0_3.8.2-1ubuntu2_i386.deb
You could try to get the latest {\f1\fs20 .deb} from @https://launchpad.net/ubuntu/vivid/i386/libsqlite3-0 \line If you want to dowwnload and install manually a {\f1\fs20 .deb} for {\f1\fs20 x86}, please install both {\i i386} and {\i amd64} revisions with the same exact version at once, otherwise {\f1\fs20 dpkg} will complain.
If it may be of any help, here are the static dependencies listed on a running 64-bit Ubuntu system, on a {\i CrossKylix} compiled executable:
$ user@server:~$ ldd Test
$       linux-gate.so.1 =>  (0xf77be000)
$       libz.so.1 => /usr/lib32/libz.so.1 (0xf779b000)
$       librt.so.1 => /lib/i386-linux-gnu/librt.so.1 (0xf7792000)
$       libpthread.so.0 => /lib/i386-linux-gnu/libpthread.so.0 (0xf7775000)
$       libdl.so.2 => /lib/i386-linux-gnu/libdl.so.2 (0xf7770000)
$       libc.so.6 => /lib/i386-linux-gnu/libc.so.6 (0xf75c1000)
$       /lib/ld-linux.so.2 (0xf77bf000)
As you can see, there is a very few dependencies - then same as FPC's executable in fact, with the addition of the external {\f1\fs20 libsqlite3.so.0}, which is statically linked to FPC's version.
:  Running Kylix 32-bit executables on 64-bit Linux
For Ubuntu versions above 13.10, if you installed a 64-bit distribution, 32-bit executables - as generated by {\i CrossKylix} -  may not be recognized by the system. Of course, we recommend using FPC (cross-)compiler, and build your executable natively for the {\f1\fs20 x86_64-linux} target.
In order to install the 32-bit libraries needed by {\i mORMot} 32-bit executables compiled by {\i Kylix} on {\i Linux}, please execute:
$ sudo apt-get install lib32z1 lib32ncurses5 lib32bz2-1.0
If you want {\f1\fs20 SynCrtSock.pas} to be able to handle {\f1\fs20 https://} on a 64-bit system - e.g. if you want to run the {\f1\fs20 TestSQL3} regression tests which download some {\f1\fs20 json} reference file over {\f1\fs20 https} - you will need also to install {\f1\fs20 @*libcurl@} (and {\f1\fs20 OpenSSL}) in 32-bit, as such:
$ sudo apt-get install libcurl3:i386
If it may be for any help, here are the static dependencies listed on a running 64-bit Ubuntu system, on a {\i FPC 3.2} compiled executable:
$user@xubuntu:~/lib/SQLite3/fpc/i386-linux$ ldd TestSQL3
$   linux-gate.so.1 =>  (0xb774c000)
$   libpthread.so.0 => /lib/i386-linux-gnu/libpthread.so.0 (0xb7718000)
$   libz.so.1 => /lib/i386-linux-gnu/libz.so.1 (0xb76fe000)
$   libdl.so.2 => /lib/i386-linux-gnu/libdl.so.2 (0xb76f8000)
$   libc.so.6 => /lib/i386-linux-gnu/libc.so.6 (0xb7549000)
$   /lib/ld-linux.so.2 (0xb774d000)
There is almost no dependency: installing a {\i mORMot} server under {\i Linux} is just as simple as copying an executable on a minimal blank {\i Linux} server. You do not need any LAMP runtime, virtual machine, installing other services, or execution environment.\line Of course, you may better add a reverse proxy like {\f1\fs20 nginx} in front of your {\i mORMot} servers when connected on the Internet, but for a @*cloud@-based solution, or a self-hosted office server, software requirements are pretty low.
: Upgrading from a 1.17 revision
If you are upgrading from an older revision of the framework, your own source code should be updated.
For instance, some units where renamed, and some breaking changes introduced by enhanced features. As a consequence, a direct update is not possible.
To properly upgrade to the latest revision:
1. Erase or rename your whole previous {\f1\fs20 #\\mORMot} directory.
2. Download latest 1.18 revision files as stated just above.
3. Change your references to {\i mORMot} units:
- Add in your uses clause {\f1\fs20 SynTests.pas} if you use testing features;
- Add in your uses clause {\f1\fs20 SynLog.pas} if you use logging features;
- Rename in your uses clauses any {\f1\fs20 SQLite3Commons} reference into {\f1\fs20 mORMot.pas};
- Rename in your uses clauses any {\f1\fs20 SQLite3} reference into {\f1\fs20 mORMotSQLite3.pas};
- Rename in your uses clauses any other {\f1\fs20 SQlite3*} reference into {\f1\fs20 mORMot*};
- Add in one of your uses clause a reference to the {\f1\fs20 SynSQLite3Static.pas} unit (for {\i Win32} or {\i Linux}).
4. Consult the units' headers about 1.18 for breaking changes, mainly:
- Introducing {\f1\fs20 @*TID@ = type Int64} as {\f1\fs20 TSQLRecord.ID} @*primary key@, {\f1\fs20 TIDDynArray} as an array, and {\f1\fs20 @*TRecordReference@} now declared as {\f1\fs20 Int64} instead of plain {\f1\fs20 PtrInt} / {\f1\fs20 integer};
- Renamed {\f1\fs20 Iso8601} low-level structure as {\f1\fs20 @*TTimeLogBits@};
- {\f1\fs20 TJSONSerializerCustomWriter} and {\f1\fs20 TJSONSerializerCustomReader} callbacks changed;
- {\f1\fs20 TSQLRestServerCallBackParams} which is replaced by the {\f1\fs20 @*TSQLRestServerURIContext@} class;
- {\f1\fs20 rmJSON*} enumerates replaced by {\f1\fs20 TSQLRestRoutingREST} and {\f1\fs20 TSQLRestRoutingJSON_RPC} classes;
- Changed '{\f1\fs20 ¤}' into '{\f1\fs20 ~}' character for {\f1\fs20 mORMoti18n.pas} (formerly {\f1\fs20 SQlite3i18n.pas}) language files.
Most of those changes will be easily identified at compile time. But a quick code review, and proper regression tests at application level is worth considering.
Feel free to get support from our forum, if needed.

[SAD-SynFile]
SourcePath=Lib\SQLite3\Samples\MainDemo
IncludePath=Lib;Lib\SQLite3;Lib\SQLite3\Samples\MainDemo
SourceFile=SynFile.dpr
SourceIgnoreSymbol=select,check,open,connect,send,sqlite3,mORMot,JavaScript,cypher,execute,cache
;SourceIgnoreSymbolByUnit=FileTables
Version=1.18
TitleOffset=0
DisplayName=Main SynFile Demo

:50SynFile application
%cartoon01.png
This sample application is a simple database tool which stores text content and files into the database, in both clear and "safe" manner. Safe records are stored using {\i AES/SHA} 256-bit encryption. There is an {\i Audit Trail} table for tracking the changes made to the database.
This document will follow the application architecture and implementation, in order to introduce the reader to some main aspects of the Framework:
- General architecture - see @7@;
- Database design - see @13@;
- User Interface generation.
We hope this part of the @SAD@ will be able to be a reliable guideline for using our framework for your own projects.
\page
: General architecture
According to the Multi-tier architecture, some units will define the three layers of the {\i SynFile} application:
{\b Database Model}
First, the database tables are defined as regular {\i Delphi} classes, like a true @*ORM@ framework. Classes are translated to database tables. @*Published properties@ of these classes are translated to table fields. No external configuration files to write - only {\i Delphi} code. Nice and easy. See @!TSQLFile,TSQLMemo,TSQLData,TSQLSafeMemo,TSQLSafeData,TSQLAuditTrail!Lib\SQLite3\Samples\MainDemo\FileTables.pas@ unit.
This unit is shared by both client and server sides, with a shared data model, i.e. a {\f1\fs20 @*TSQLModel@} class instance, describing all ORM tables/classes.
It contains also internal event descriptions, and actions, which will be used to describe the software UI.
{\b Business Logic}
The {\i server side} is defined in a dedicated class, which implements an automated Audit Trail, and a @*service@ named "Event" to easily populate the Audit Trail from the Client side. See @!TFileServer!Lib\SQLite3\Samples\MainDemo\FileServer.pas@ unit.
The {\i client side} is defined in another class, which is able to communicate with the server, and fill/update/delete/add the database content playing with classes instances. It's also used to call the Audit Trail related service, and create the reports. See @!TFileClient!Lib\SQLite3\Samples\MainDemo\FileClient.pas@ unit.
Client-Server logic will be detailed in the next paragraph.
{\b Presentation Layer}
The main form of the Client is void, if you open its {\f1\fs20 FileMain.dfm} file. All the User Interface is created by the framework, dynamically from the database model and some constant values and enumeration types (thanks to {\i Delphi} @*RTTI@) as defined in @!TFileRibbonTabParameters.Actions!Lib\SQLite3\Samples\MainDemo\FileTables.pas@ unit (the first one, which defines also the classes/tables).
It's main method is {\f1\fs20 TMainForm.ActionClick}, which will handle the actions, triggered when a button is pressed.
The @**report@s use {\i GDI+} for anti-aliased drawing, can be zoomed and saved as @*pdf@ or text files.
The last @!TEditForm!Lib\SQLite3\Samples\MainDemo\FileEdit.pas@ unit is just the form used for editing the data. It also performs the encryption of "safe memo" and "safe data" records, using our @!TAESFull.EncodeDecode!Lib\SynCrypto.pas@ unit. It will @*AES-NI@ hardware instructions, if available, so will be very fast, even for big content.
You'll discover how the @*ORM@ plays its role here: you change the data, just like changing any class instance properties.
It also uses our @!SaveAsRawByteString!Lib\SynGdiPlus.pas@ unit to create thumbnails of any picture ({\f1\fs20 emf+jpg+tif+gif+bmp}) of data inserted in the database, and add a @*BLOB@ data field containing these thumbnails.
\page
:128 Database design
The @!TSQLFile,TSQLMemo,TSQLData,TSQLSafeMemo,TSQLSafeData,TSQLAuditTrail!Lib\SQLite3\Samples\MainDemo\FileTables.pas@ unit is implementing all {\f1\fs20 @*TSQLRecord@} child classes, able to create the database tables, using the @*ORM@ aspect of the framework - see @13@. The following class hierarchy was designed:
\graph HierSynFileRecord SynFile TSQLRecord classes hierarchy
\TSQLAuditTrail\TSQLRecord
\TSQLSafeMemo\TSQLData
\TSQLData\TSQLFile
\TSQLSafeData\TSQLData
\TSQLMemo\TSQLFile
\TSQLFile\TSQLRecordSigned
\TSQLRecordSigned\TSQLRecord
\
Most common @*published properties@ (i.e. {\f1\fs20 Name, Created, Modified, Picture, KeyWords}) are taken from the {\f1\fs20 TSQLFile} abstract parent class. It's called "{\i abstract}", not in the current {\i Delphi} @*OOP@ terms, but as a class with no "real" database table associated. It was used to defined the properties only once, without the need of writing the private variables nor the getter/setter for children classes. Only {\f1\fs20 TSQLAuditTrail} won't inherit from this parent class, because it's purpose is not to contain data, but just some information.
The database itself will define {\f1\fs20 TSQLAuditTrail, TSQLMemo, TSQLData, TSQLSafeMemo}, and {\f1\fs20 TSQLSafeData} classes. They will be stored as {\i AuditTrail, Memo, Data, SafeMemo} and {\i SafeData} tables in the {\i @*SQlite3@} database (the table names are extract from the class name, trimming the left '{\f1\fs20 TSQL}' characters).
Here is this common ancestor type declaration:
!  TSQLFile = class(TSQLRecordSigned)
!  public
!    fName: RawUTF8;
!    fModified: TTimeLog;
!    fCreated: TTimeLog;
!    fPicture: TSQLRawBlob;
!    fKeyWords: RawUTF8;
!  published
!    property Name: RawUTF8 read fName write fName;
!    property Created: TTimeLog read fCreated write fCreated;
!    property Modified: TTimeLog read fModified write fModified;
!    property Picture: TSQLRawBlob read fPicture write fPicture;
!    property KeyWords: RawUTF8 read fKeyWords write fKeyWords;
!  end;
Sounds like a regular {\i Delphi} class, doesn't it? The only fact to be noticed is that it does not inherit from a {\f1\fs20 @*TPersistent@} class, but from a {\f1\fs20 @*TSQLRecord@} class, which is the parent object type to be used for our ORM. The {\f1\fs20 TSQLRecordSigned} class type just defines some {\f1\fs20 Signature} and {\f1\fs20 SignatureTime} additional properties, which will be used here for handling digital signing of records.
Here follows the {\i Delphi} code written, and each corresponding database field layout of each registered class:
!  TSQLMemo = class(TSQLFile)
!  public
!    fContent: RawUTF8;
!  published
!    property Content: RawUTF8 read fContent write fContent;
!  end;
\graph DBMemo Memo Record Layout
rankdir=LR;
node [shape=Mrecord];
struct1 [label="ID : TID|Content : RawUTF8|Created : TTimeLog|KeyWords : RawUTF8|Modified : TTimeLog|Name : RawUTF8|Picture : TSQLRawBlob|Signature : RawUTF8|SignatureTime: TTimeLog"];
\
!  TSQLData = class(TSQLFile)
!  public
!    fData: TSQLRawBlob;
!  published
!    property Data: TSQLRawBlob read fData write fData;
!  end;
\graph DBData Data Record Layout
rankdir=LR;
node [shape=Mrecord];
struct1 [label="ID : TID|Data : TSQLRawBlob|Created : TTimeLog|KeyWords : RawUTF8|Modified : TTimeLog|Name : RawUTF8|Picture : TSQLRawBlob|Signature : RawUTF8|SignatureTime: TTimeLog"];
\
!  TSQLSafeMemo = class(TSQLData);
\graph DBSafeMemo SafeMemo Record Layout
rankdir=LR;
node [shape=Mrecord];
struct1 [label="ID : TID|Data : TSQLRawBlob|Created : TTimeLog|KeyWords : RawUTF8|Modified : TTimeLog|Name : RawUTF8|Picture : TSQLRawBlob|Signature : RawUTF8|SignatureTime: TTimeLog"];
\
!  TSQLSafeData = class(TSQLData);
\graph DBSafeData SafeData Record Layout
rankdir=LR;
node [shape=Mrecord];
struct1 [label="ID : TID|Data : TSQLRawBlob|Created : TTimeLog|KeyWords : RawUTF8|Modified : TTimeLog|Name : RawUTF8|Picture : TSQLRawBlob|Signature : RawUTF8|SignatureTime: TTimeLog"];
\
You can see that {\f1\fs20 TSQLSafeMemo} and {\f1\fs20 TSQLSafeData} are just a direct sub-class of {\f1\fs20 TSQLData} to create "{\i SafeMemo}" and "{\i SafeData}" tables with the exact same fields as the "{\i Data}" table. Since they were declared as {\f1\fs20 class(TSQLData)}, they are some new class type,
Then the latest class is not inheriting from {\f1\fs20 TSQLFile}, because it does not contain any user data, and is used only as a log of all actions performed using {\i SynFile}:
!  TSQLAuditTrail = class(TSQLRecord)
!  protected
!    fStatusMessage: RawUTF8;
!    fStatus: TFileEvent;
!    fAssociatedRecord: TRecordReference;
!    fTime: TTimeLog;
!  published
!    property Time: TTimeLog read fTime write fTime;
!    property Status: TFileEvent read fStatus write fStatus;
!    property StatusMessage: RawUTF8 read fStatusMessage write fStatusMessage;
!    property AssociatedRecord: TRecordReference read fAssociatedRecord write fAssociatedRecord;
!  end;
\graph DBAuditTrail AuditTrail Record Layout
rankdir=LR;
node [shape=Mrecord];
struct1 [label="ID : TID|AssociatedRecord : TRecordReference|Status : TFileEvent|StatusMessage : RawUTF8|Time : TTimeLog"];
\
The {\f1\fs20 AssociatedRecord} property was defined as {\f1\fs20 @*TRecordReference@}. This special type (mapped as an INTEGER field in the database) is able to define a "one to many" relationship with ANY other record of the database model.
- If you want to create a "one to many" relationship with a particular table, you should define a property with the corresponding {\f1\fs20 @*TSQLRecord@} sub-type (for instance, if you want to link to a particular {\i SafeData} row, define the property as {\f1\fs20 AssociatedData: TSQLSafeData;}) - in this case, this will create an INTEGER field in the database, holding the {\i RowID} value of the associated record (and this field content will be filled with {\f1\fs20 pointer(RowID)} and not with a real {\f1\fs20 TSQLSafeData} instance).
- Using a {\f1\fs20 TRecordReference} type won't link to a particular table, but any table of the database model: it will store in its associated INTEGER database field not only the {\i RowID} of the record, but also the table index as registered at {\f1\fs20 @*TSQLModel@} creation. In order to access this {\f1\fs20 AssociatedRecord} property content, you could use either {\f1\fs20 @*TSQLRest@. Retrieve(AssociatedRecord)} to get the corresponding record instance, or typecast it to {\f1\fs20 @*RecordRef@} wrapper structure to easily retrieve or set the associated table and {\i RowID}. You could also use the {\f1\fs20 TSQLRecord. RecordReference(Model)} method in order to get the value corresponding to an existing {\f1\fs20 TSQLRecord} instance.
According to the @*MVC@ model - see @10@ - the framework expect a common database model to be shared between client and server. A common function has been defined in the @!CreateFileModel!Lib\SQLite3\Samples\MainDemo\FileTables.pas@ unit, as such:
!function CreateFileModel(Owner: TSQLRest): TSQLModel;
We'll see later its implementation. Just note for the moment that it will register the {\f1\fs20 TSQLAuditTrail, TSQLMemo, TSQLData, TSQLSafeMemo}, and {\f1\fs20 TSQLSafeData} classes as part of the database model. The order of the registration of those classes will be used for the {\f1\fs20 AssociatedRecord: TRecordReference} field of {\f1\fs20 TSQLAuditTrail} - e.g. a {\f1\fs20 TSQLMemo} record will be identified with a table index of 1 in the {\f1\fs20 RecordReference} encoded value. So it's mandatory to NOT change this order in any future modification of the database schema, without providing any explicit database content conversion mechanism.
Note that all above graphs were created directly from our {\i @*SynProject@} tool, which is able to create custom graphs from the application source code it parsed.
\page
: Client Server implementation
{\i Server}-side is implemented in unit {\f1\fs20 FileServer}, with the following class:
!  TFileServer = class(TSQLRestserverDB)
!   (...)
!    Server: TSQLHttpServer;
!    procedure AddAuditTrail(aEvent: TFileEvent; const aMessage: RawUTF8='';
!      aAssociatedRecord: TRecordReference=0);
!    function OnDatabaseUpdateEvent(Sender: TSQLRestServer;
!      Event: TSQLEvent; aTable: TSQLRecordClass; aID: TID): boolean;
!  published
!    procedure Event(Ctxt: TSQLRestServerURIContext);
!  end;
As stated above, it inherits from {\f1\fs20 TSQLRestserverDB} to define a RESTful ORM based on the {\i @*SQLite3@} database engine, and define a custom method-based service named {\f1\fs20 Event}.
The class constructor creates the whole server-side logic, following the shared data {\i Model} as defined in the {\f1\fs20 FileTables} unit:
!constructor TFileServer.Create;
!begin
!  inherited Create(CreateFileModel(self),ChangeFileExt(paramstr(0),'.db3'));
!  CreateMissingTables(ExeVersion.Version.Version32);
!  Server := TSQLHttpServer.Create(SERVER_HTTP_PORT,self,'+',useHttpApiRegisteringURI);
!  AddAuditTrail(feServerStarted);
!  OnUpdateEvent := OnDatabaseUpdateEvent;
!end;
A dedicated {\f1\fs20 TSQLHttpServer} instance is initialized to publish the {\f1\fs20 TFileServer} content over HTTP.
Automatic logging in the {\i Audit Trail} table of most database changes is performed via the {\f1\fs20 OnDatabaseUpdateEvent} callback.
{\i Client}-side is implemented in unit {\f1\fs20 FileClient}, with the following class:
!  TFileClient = class(TSQLHttpClient)
!  public
!    (...)
!    procedure AddAuditTrail(aEvent: TFileEvent; aAssociatedRecord: TSQLRecord);
!  end;
You'll see that @*BLOB@ fields are handled just like other fields, even if they use their own @*REST@ful GET/PUT dedicated URI (they are not @*JSON@ encoded, but transmitted as raw data, to save bandwidth and maintain the RESTful model). The framework handles it for you, thanks to its ORM orientation, in the {\f1\fs20 TFileClient} constructor:
!constructor TFileClient.Create(const aServer: AnsiString);
!begin
!  inherited Create(aServer,SERVER_HTTP_PORT,CreateFileModel(self));
!  ForceBlobTransfert := true;
!end;
Here, we set {\f1\fs20 @*ForceBlobTransfert@ := true}, since by default all BLOB fields won't be transmitted by {\f1\fs20 TSQLRestClientURI}, whereas our simple application expect them to be always available.
The same data {\i Model},  as defined in the {\f1\fs20 FileTables} unit, is used also on the client side.
On both sides, a {\f1\fs20 AddAuditTrail()} is defined, to allow direct logging to the internal {\f1\fs20 TSQLAuditTrail} table. From the client, it uses the {\f1\fs20 Event} method-based service to perform the remote action:
!procedure TFileClient.AddAuditTrail(aEvent: TFileEvent;
!  aAssociatedRecord: TSQLRecord);
!begin
!  if aAssociatedRecord=nil then
!    CallBackGetResult('Event',['event',ord(aEvent)]) else
!    with aAssociatedRecord do
!      CallBackGetResult('Event',['event',ord(aEvent)],RecordClass,ID);
!end;
\page
:31 User Interface generation
You could of course design your own User Interface without our framework. That is, this is perfectly feasible to use only the @*ORM@ part of it. For instance, it should be needed to develop @*AJAX@ applications using its @*REST@ful model - see @9@ - since such a feature is not yet integrated to our provided source code.
But for producing easily applications, the framework provides a mechanism based on both ORM description and @*RTTI@ compiler-generated information in order to create most User Interface by code.
It is able to generated a Ribbon-based application, in which each table is available via a Ribbon tab, and some actions performed to it.
So the framework will need to know:
- Which tables must be displayed;
- Which actions should be associated with each table;
- How the User Interface should be customized (e.g. hint texts, grid layout on screen, reporting etc...);
- How generic automated edition, using the @!TRecordEditForm!Lib\SQLite3\mORMotUIEdit.pas@ unit, is to be generated.
To this list could be added an integrated event feature, which can be linked to actions and custom status, to provide a centralized handling of user-level @*log@ing (as used e.g. in the {\i SynFile} {\f1\fs20 TSQLAuditTrail} table) - please do not make confusion between this user-level logging and technical-level logging using {\f1\fs20 TSynLog} and {\f1\fs20 TSQLLog} classes and "families" - see @16@.
:  Rendering
The current implementation of the framework User Interface generation handles two kind of rendering:
- Native VCL components;
- Proprietary @*TMS@ components.
You can select which set of components are used, by defining - globally to your project (i.e. in the {\i Project/Options/Conditionals} menu) - the {\f1\fs20 USETMSPACK} conditional. If it is not set (which is by default), it will use VCL components.
The native VCL components will use native Windows API components. So the look and feel of the application will vary depending on the Windows version it is running on. For instance, the resulting screen will be diverse if the application is run under Windows 2000, XP, Vista and Seven. The "ribbon" as generated with VCL components has most functionalities than the Office 2007/2010 ribbon, but will have a very diverse layout.
The TMS components will have the same rendering whatever the Windows it's running on, and will display a "ribbon" very close to the official Office 2007/2010 version.
Here are some PROs and CONs about both solutions:
|%35%30%30
|\b Criteria|VCL|TMS\b0
|Rendering|Basic|Sophisticated
|OS version|Variant|Constant
|Ribbon look|Unusual|Office-like
|Preview button & Shortcuts|None by default|Available
|Extra Price|None|High
|GPL ready|Yes|No
|Office UI Licensing|N/A|Required
|EXE size|Smaller|Bigger
|%
It's worth saying that the choice of one or other component set could be changed on request. If you use the generic components as defined in {\f1\fs20 mORMotToolBar} (i.e. the {\f1\fs20 TSynForm, TSynToolBar, TSynToolButton, TSynPopupMenu, TSynPage, TSynPager, TSynBodyPager} and {\f1\fs20 TSynBodyPage} classes) and {\f1\fs20 SynTaskDialog} (for {\f1\fs20 TSynButton}) in your own code, the {\f1\fs20 USETMSPACK} conditional will do all the magic for you.
The {\i Office UI licensing program} was designed by {\i Microsoft} for software developers who wish to implement the Office UI as a software component and/or incorporate the Office UI into their own applications. If you use TMS ribbon, it does not require any more acceptance of the Office UI License terms - see at @http://msdn.microsoft.com/en-us/office/aa973809.aspx
If you want to design your user interface using a Office 2007/2010 ribbon look, please take a look at those official guidelines: @http://msdn.microsoft.com/en-us/library/cc872782.aspx
Here is the screen content, using the TMS components:
%synfiletms.png
And here is the same application compiled using only VCL components, available from {\i Delphi} 6 up to the latest {\i Delphi} version:
%synfilevcl.png
We did not use yet the Ribbon component as was introduced in {\i Delphi} 2009. Its action-driven design won't make it easy to interface with the event-driven design of our User Interface handling, and we have to confess that this component has rather bad reputation (at least in the {\i Delphi} 2009 version). Feel free to adapt our Open Source code to use it - we'll be very pleased to release a new version supporting it, but we don't have time nor necessity to do it by ourself.
:  Enumeration types
A list of available actions should be defined, as an enumeration type:
!  TFileAction = (
!    faNoAction, faMark, faUnmarkAll, faQuery, faRefresh, faCreate,
!    faEdit, faCopy, faExport, faImport, faDelete, faSign, faPrintPreview,
!    faExtract, faSettings );
Thanks to the {\i Delphi} @*RTTI@, and "{\i Un @*Camel@ Casing}", the following list will generate a set of available buttons on the User Interface, named "Mark", "Unmark all", "Query", "Refresh", "Create", "Edit", "Copy", "Export", "Import", "Delete", "Sign", "Print preview", "Extract" and "Settings". Thanks to the @!TLanguageFile.Translate!Lib\SQLite3\mORMoti18n.pas@ unit (responsible of application @*i18n@) and the {\f1\fs20 TLanguageFile. Translate} method, it could be translated on-the-fly from English into the current desired language, before display on screen or report creation.
See both above screen-shots to guess how the button captions match the enumeration names - i.e. @%synfilevcl.png@ and @%synfilevcl.png@.
A list of events, as used for the {\f1\fs20 TSQLAuditTrail} table, was also defined. Some events reflect the change made to the database rows (like {\f1\fs20 feRecordModified}), or generic application status (like {\f1\fs20 feServerStarted}):
!  TFileEvent = (
!    feUnknownState, feServerStarted, feServerShutdown,
!    feRecordCreated, feRecordModified, feRecordDeleted,
!    feRecordDigitallySigned, feRecordImported, feRecordExported );
In the grid and the reports, @*RTTI@ and "{\i uncamelcasing}" will be used to display this list as regular text, like "{\i Record digitally signed}", and translated to the current language, if necessary.
:  ORM Registration
The User Interface generation will be made by creating an array of objects inheriting from the {\f1\fs20 TSQLRibbonTabParameters} type.
Firstly, a custom object type is defined, associating
!  TFileRibbonTabParameters = object(TSQLRibbonTabParameters)
!    /// the SynFile actions
!    Actions: TFileActions;
!  end;
Then a constant array of such objects is defined:
!const
!FileTabs: array[0..4] of TFileRibbonTabParameters = (
!(Table: TSQLAuditTrail;
! Select: 'Time,Status,StatusMessage'; Group: GROUP_MAIN;
! FieldWidth: 'gIZ'; ShowID: true; ReverseOrder: true; Layout: llClient;
! Actions: [faDelete,faMark,faUnmarkAll,faQuery,faRefresh,faPrintPreview,faSettings]),
!(Table: TSQLMemo;
! Select: DEF_SELECT; Group: GROUP_CLEAR; FieldWidth: 'IddId'; Actions: DEF_ACTIONS),
!(Table: TSQLData;
! Select: DEF_SELECT; Group: GROUP_CLEAR; FieldWidth: 'IddId'; Actions: DEF_ACTIONS_DATA),
!(Table: TSQLSafeMemo;
! Select: DEF_SELECT; Group: GROUP_SAFE; FieldWidth: 'IddId';  Actions: DEF_ACTIONS),
!(Table: TSQLSafeData;
! Select: DEF_SELECT; Group: GROUP_SAFE; FieldWidth: 'IddId';  Actions: DEF_ACTIONS_DATA));
The {\f1\fs20 Table} property will map the @*ORM@ class to the User Interface ribbon tab. A custom CSV list of fields should be set to detail which database columns must be displayed on the grids and the reports, in the {\f1\fs20 Select} property. Each ribbon tab could contain one or more {\f1\fs20 @*TSQLRecord@} table: the {\f1\fs20 Group} property is set to identify on which ribbon group it should be shown. The grid column widths are defined as a {\f1\fs20 FieldWidth} string in which each displayed field length mean is set with one char per field (A=first {\f1\fs20 Select} column,Z=26th column) - lowercase character will center the field data. For each table, the available actions are also set, and will be used to create the possible buttons to be shown on the ribbon toolbars (enabling or disabling a button is to be done at runtime).
Note that this array definition uses some previously defined individual constants (like {\f1\fs20 DEF_SELECT}, {\f1\fs20 DEF_ACTIONS_DATA} or {\f1\fs20 GROUP_SAFE}. This is a good practice, and could make code maintenance easier later on.
:  Main window
Once all this ORM and action information is available, the {\f1\fs20 FileMain} unit defines the following class to generate the expected ribbon-based User Interface:
!  TMainForm = class(TSynForm)
!    ImageList32: TImageList;
!    ImageList16: TImageList;
!    procedure FormCreate(Sender: TObject);
!    procedure FormShow(Sender: TObject);
!  private
!{$ifdef DEBUGINTERNALSERVER}
!    Server: TFileServer;
!{$endif}
!  protected
!    procedure ActionClick(Sender: TObject; const RecordClass: TSQLRecordClass;
!      ActionValue: integer);
!    procedure HelpClick(Sender: TObject);
!    procedure ListDblClick(Sender: TObject);
!    procedure WMRefreshTimer(var Msg: TWMTimer); message WM_TIMER;
!    function Edit(aRec: TSQLFile; const aTitle: string; aReadOnly: boolean): boolean;
!  public
!    Client: TFileClient;
!    Ribbon: TFileRibbon;
!    destructor Destroy; override;
!  end;
The following method will do all the initialization, from the given Data and Presentation Models:
!procedure TMainForm.FormCreate(Sender: TObject);
!var P: integer;
!begin
!{$ifdef DEBUGINTERNALSERVER}
!  try
!    Server := TFileServer.Create;
!  except
!    on E: Exception do begin
!      ShowException(E);
!      exit;
!    end;
!  end;
!{$endif}
!  LoadImageListFromEmbeddedZip(ImageList32,'buttons.bmp');
!  ImageListStretch(ImageList32,ImageList16);
!  Client := TFileClient.Create('localhost');
!  Client.OnIdle := TLoginForm.OnIdleProcessForm;
!  Ribbon := TFileRibbon.Create(self, nil, nil, ImageList32, ImageList16,
!    Client, ALL_ACCESS_RIGHTS, nil, Client.OnSetAction, sFileActionsToolbar,
!    sFileActionsHints, nil, ActionClick, integer(faRefresh), 1, false,
!    length(FileTabs), @FileTabs[0], sizeof(FileTabs[0]),
!    sFileTabsGroup, ',BannerData,BannerSafe',true);
!  Ribbon.ToolBar.Caption.Caption := Caption;
!  Ribbon.ToolBar.HelpButton.OnClick := HelpClick;
!  for P := 0 to high(Ribbon.Page) do
!    with Ribbon.Page[P] do
!      if Lister<>nil then
!        Lister.Grid.OnDblClick := ListDblClick;
!end;
Even if a real application may be truly Client-Server, we define a stand-alone mode. That is, a {\f1\fs20 TFileServer} instance is instantiated within the main application execution. Just un-define the {\f1\fs20 DEBUGINTERNALSERVER} conditional if you want a "pure client" version of the application - in this case, a stand-alone server shall be running.
All the ORM and actions defined in the {\f1\fs20 FileTables} unit are used to initialize the {\f1\fs20 TFileRibbon} content in the {\f1\fs20 Ribbon} field which will be the main entry point of all User Interface process.
The {\f1\fs20 ActionClick()} method is the main entry point of the application, and is called when the User clicks on any ribbon button. It is just a {\f1\fs20 case Action of ...} switch instruction, handling each {\f1\fs20 TFileAction} event as expected.
The {\f1\fs20 Edit()} method will allow edition of a given record fields, via the separated {\f1\fs20 TEditForm} window, as defined in {\f1\fs20 FileEdit} unit. We won't use the auto-generated window from RTTI in this case, since we expect a dedicated process to attach a picture to the corresponding {\f1\fs20 TSQLFile} item.
The {\f1\fs20 ListDblClick()} method will process any double click on the list to edit the corresponding item ({\f1\fs20 faEdit} action), or go to the record an audit trail row refers to, using a convenient local {\f1\fs20 @*RecordRef@} wrapper variable:
!procedure TMainForm.ListDblClick(Sender: TObject);
!var P: TSQLRibbonTab;
!    ref: RecordRef;
!begin
!  P := Ribbon.GetActivePage;
!  if P<>nil then
!    if P.Table=TSQLAuditTrail then begin
!      if P.Retrieve(Client,P.List.Row) then begin
!        ref.Value := TSQLAuditTrail(P.CurrentRecord).AssociatedRecord;
!        Ribbon.GotoRecord(ref.Table(Client.Model),ref.ID);
!      end;
!    end else
!    ActionClick(Sender,P.Table,ord(faEdit));
!end;
The {\f1\fs20 WMRefreshTimer()} method will just transmit any {\f1\fs20 WM_TIMER} event to the ribbon process, in order to handle automatic refresh of the content, following the {\i stateless} approach of our RESTful framework:
!procedure TMainForm.WMRefreshTimer(var Msg: TWMTimer);
!begin
!  Ribbon.WMRefreshTimer(Msg);
!end;
You probably noticed that {\f1\fs20 Client.OnIdle} is set in the {\f1\fs20 FormCreate} method to map the {\f1\fs20 TLoginForm.OnIdleProcessForm} callback. This will let the HTTP client class to use a background thread for all communication, instead of blocking the main application thread. The main User Interface will still be responsive (since {\f1\fs20 OnIdleProcessForm} will call {\f1\fs20 Application.ProcessMessages}), and change the cursor to {\f1\fs20 crHourGlass} in case of a slow request, or even display a temporary pop-up with {\i "Please wait..."} if the network is really slow, and the request takes more than 2 seconds (all those notification parameters can be changed in {\f1\fs20 mORMotUILogin.pas}).
\page
:67 Report generation
The following {\f1\fs20 CreateReport} method is overridden in @!TFileRibbon.CreateReport!Lib\SQLite3\Samples\MainDemo\FileClient.pas@:
!  /// class used to create the User interface
!  TFileRibbon = class(TSQLRibbon)
!  public
!    /// overridden method used customize the report content
!    procedure CreateReport(aTable: TSQLRecordClass; aID: TID; aReport: TGDIPages;
!      AlreadyBegan: boolean=false); override;
!  end;
The reporting engine in the framework is implemented via the {\f1\fs20 TGDIPages} class, defined in the @!TGDIPages!Lib\SQLite3\mORMotReport.pas@:
- Data is drawn in memory, they displayed or printed as desired;
- High-level reporting methods are available (implementing tables, columns, titles and such), but you can have access to a {\f1\fs20 TCanvas} property which allows any possible content generation via standard VCL methods;
- Allow preview (with anti-aliased drawing via GDI+) and printing;
- Direct export as {\f1\fs20 .txt} or {\f1\fs20 .@*pdf@} file;
- Handle bookmark, outlines and links inside the document.
By default, the {\f1\fs20 CreateReport} method of {\f1\fs20 TSQLRibbon} will write all editable fields value to the content.
The method is overridden by the following code:
!procedure TFileRibbon.CreateReport(aTable: TSQLRecordClass; aID: TID; aReport: TGDIPages;
!  AlreadyBegan: boolean=false);
!var Rec: TSQLFile;
!    Pic: TBitmap;
!    s: string;
!    PC: PChar;
!    P: TSQLRibbonTab;
!begin
!  with aReport do
!  begin
!    // initialize report
!    Clear;
!    BeginDoc;
!    Font.Size := 10;
!    if not aTable.InheritsFrom(TSQLFile) then
!      P := nil else
!      P := GetActivePage;
!    if (P=nil) or (P.CurrentRecord.ID<>aID) or (P.Table<>aTable) then
!    begin
!      inherited; // default handler
!      exit;
!    end;
!    Rec := TSQLFile(P.CurrentRecord);
!    Caption := U2S(Rec.fName);
The report is cleared, and {\f1\fs20 BeginDoc} method is called to start creating the internal canvas and band positioning. The font size is set, and parameters are checked against expected values. Then the current viewed record is retrieved from {\f1\fs20 GetActivePage. CurentRecord}, and the report caption is set via the record {\f1\fs20 Name} field.
!    // prepare page footer
!    SaveLayout;
!    Font.Size := 9;
!    AddPagesToFooterAt(sPageN,LeftMargin);
!    TextAlign := taRight;
!    AddTextToFooterAt('SynFile  https://synopse.info - '+Caption,RightMarginPos);
!    RestoreSavedLayout;
Page footer are set by using two methods:
- {\f1\fs20 AddPagesToFooterAt} to add the current page number at a given position (here the left margin);
- {\f1\fs20 AddTextToFooterAt} to add some custom text at a given position (here the right margin, after having changed the text alignment into right-aligned).
Note that {\f1\fs20 SaveLayout/RestoreSavedLayout} methods are used to modify temporary the current font and paragraph settings for printing the footer, then restore the default settings.
!    // write global header at the beginning of the report
!    DrawTitle(P.Table.CaptionName+' : '+Caption,true);
!    NewHalfLine;
!    AddColumns([6,40]);
!    SetColumnBold(0);
!    if Rec.SignatureTime<>0 then
!    begin
!      PC := Pointer(Format(sSignedN,[Rec.SignedBy,Iso2S(Rec.SignatureTime)]));
!      DrawTextAcrossColsFromCSV(PC,$C0C0FF);
!    end;
!    if Rec.fCreated<>0 then
!      DrawTextAcrossCols([sCreated,Iso2S(Rec.fCreated)]);
!    if Rec.fModified<>0 then
!      DrawTextAcrossCols([sModified,Iso2S(Rec.fModified)]);
!    if Rec.fKeyWords='' then
!      s := sNone else
!    begin
!      s := U2S(Rec.fKeyWords);
!      ExportPDFKeywords := s;
!    end;
!    DrawTextAcrossCols([sKeyWords,s]);
!    NewLine;
!    Pic := LoadFromRawByteString(Rec.fPicture);
!    if Pic<>nil then
!    try
!      DrawBMP(Pic,0,Pic.Width div 3);
!    finally
!      Pic.Free;
!    end;
Report header is written using the following methods:
- {\f1\fs20 DrawTitle} to add a title to the report, with a black line below it (second parameter to {\f1\fs20 true}) - this title will be added to the report global outline, and will be exported as such in {\f1\fs20 .@*pdf@} on request;
- {\f1\fs20 NewHalfLine} and {\f1\fs20 NewLine} will leave some vertical gap between two paragraphs;
- {\f1\fs20 AddColumns}, with parameters set as percentages, will initialize a table with the first column content defined as bold ({\f1\fs20 SetColumnBold(0)});
- {\f1\fs20 DrawTextAcrossCols} and {\f1\fs20 DrawTextAcrossColsFromCSV} will fill a table row according to the text specified, one string per column;
- {\f1\fs20 DrawBMP} will draw a bitmap to the report, which content is loaded using the generic {\f1\fs20 LoadFromRawByteString} function implemented in @!Lib\SynGdiPlus.pas@;
- {\f1\fs20 U2S} and {\f1\fs20 Iso2S} function, as defined in @!Iso2S,U2S!Lib\SQLite3\mORMoti18n.pas@, are used for conversion of some text or {\f1\fs20 @*TTimeLog@/@*TUnixTime@} into a text formated with the current language settings (@*i18n@).
!    // write report content
!    DrawTitle(sContent,true);
!    SaveLayout;
!    Font.Name := 'Courier New';
!    if Rec.InheritsFrom(TSQLSafeMemo) then
!      DrawText(sSafeMemoContent) else
!    if Rec.InheritsFrom(TSQLMemo) then
!      DrawTextU(TSQLMemo(Rec).Content) else
!    if Rec.InheritsFrom(TSQLData) then
!    with TSQLData(Rec) do
!    begin
!      DrawTextU(Rec.fName);
!      s := PictureName(TSynPicture.IsPicture(TFileName(Rec.fName)));
!      if s<>'' then
!        s := format(sPictureN,[s]) else
!        if not Rec.InheritsFrom(TSQLSafeData) then
!          s := U2S(GetMimeContentType(Pointer(Data),Length(Data),TFileName(Rec.fName)));
!      if s<>'' then
!        DrawTextFmt(sContentTypeN,[s]);
!      DrawTextFmt(sSizeN,[U2S(KB(Length(Data)))]);
!      NewHalfLine;
!      DrawText(sDataContent);
!    end;
!    RestoreSavedLayout;
Then the report content is appended, according to the record class type:
- {\f1\fs20 DrawText}, {\f1\fs20 DrawTextU} and {\f1\fs20 DrawTextFmt} are able to add a paragraph of text to the report, with the current alignment - in this case, the font is set to '{\i Courier New}' so that it will be displayed with fixed width;
- {\f1\fs20 GetMimeContentType} is used to retrieve the exact type of the data stored in this record.
!    // set custom report parameters
!    ExportPDFApplication := 'SynFile  https://synopse.info';
!    ExportPDFForceJPEGCompression := 80;
!  end;
!end;
Those {\f1\fs20 ExportPDFApplication} and {\f1\fs20 ExportPDFForceJPEGCompression} properties (together with the {\f1\fs20 ExportPDFKeywords} are able to customize how the report will be exported into a {\f1\fs20 .pdf} file. In our case, we want to notify that {\i SynFile} generated those files, and that the header bitmap should be compressed as JPEG before writing to the file (in order to produce a small sized {\f1\fs20 .pdf}).
You perhaps did notice that textual constant were defined as {\f1\fs20 @*resourcestring@}, as such:
!resourcestring
!  sCreated = 'Created';
!  sModified = 'Modified';
!  sKeyWords = 'KeyWords';
!  sContent = 'Content';
!  sNone = 'None';
!  sPageN = 'Page %d / %d';
!  sSizeN = 'Size: %s';
!  sContentTypeN = 'Content Type: %s';
!  sSafeMemoContent = 'This memo is password protected.'#13+
!    'Please click on the "Edit" button to show its content.';
!  sDataContent = 'Please click on the "Extract" button to get its content.';
!  sSignedN = 'Signed,By %s on %s';
!  sPictureN = '%s Picture';
The @!Lib\SQLite3\mORMoti18n.pas@ unit is able to parse all those {\f1\fs20 resourcestring} from a running executable, via its {\f1\fs20 ExtractAllResources} function, and create a reference text file to be translated into any handled language.
Creating a report from code does make sense in an ORM. Since we have most useful data at hand as {\i Delphi} classes, code can be shared among all kind of reports, and a few lines of code is able to produce complex reports, with enhanced rendering, unified layout, direct internationalization and export capabilities.
Note that the {\f1\fs20 mORMotReport.pas} unit uses UTF-16 encoded string, i.e. our {\f1\fs20 SynUnicode} type, which is either {\f1\fs20 UnicodeString} since Delphi 2009, or {\f1\fs20 WideString} for older versions. {\f1\fs20 WideString} is known to have performance issues, due to use of slow BSTR API calls - so if you want to create huge reports with pre-Unicode versions of Delphi and our report engine, consider adding a reference to our {\f1\fs20 @*SynFastWideString.pas@} unit at first place of your {\f1\fs20 .dpr} uses clause, for potential huge speed enhancement. See @32@ for more details, especially the restriction of use, since it will break any attempt to use BSTR parameters with any OLE/COM object.
\page
:115 Application i18n and L10n
In computing, internationalization and localization (also spelled internationalisation and localisation) are means of adapting computer software to different languages, regional differences and technical requirements of a target market:
- {\i Internationalization} (@**i18n@) is the process of designing a software application so that it can be adapted to various languages;
- {\i Localization} (@**L10n@) is the process of adapting internationalized software for a specific region or language by adding locale-specific components and translating text, e.g. for dates display.
Our framework handles both features, via the @!Lib\SQLite3\mORMoti18n.pas@ unit. We just saw above how {\f1\fs20 @*resourcestring@} defined in the source code are retrieved from the executable and can be translated on the fly. The unit extends this to visual forms, and even captions generated from @*RTTI@ - see @5@.
The unit expects all textual content (both {\f1\fs20 resourcestring} and RTTI derived captions) to be correct English text. A list of all used textual elements will be retrieved then hashed into an unique numerical value. When a specific locale is set for the application, the unit will search for a {\f1\fs20 @*.msg@} text file in the executable folder matching the expected locale definition. For instance, it will search for {\f1\fs20 FR.msg} for translation into French.
In order to translate all the user interface, a corresponding {\f1\fs20 .msg} file is to be supplied in the executable folder. Neither the source code, nor the executable is to be rebuild to add a new language. And since this file is indeed a plain textual file, even a non developer (e.g. an end-user) is able to add a new language, starting from another {\f1\fs20 .msg}.
:  Creating the reference file
In order to begin a translation task, the {\f1\fs20 mORMoti18n.pas} unit is able to extract all textual resource from the executable, and create a reference text file, containing all English sentences and words to be translated, associated with their numerical hash value.
It will in fact:
- Extract all {\f1\fs20 resourcestring} text;
- Extract all captions generated from RTTI (e.g. from enumerations or class properties names);
- Extract all embedded {\f1\fs20 dfm} resources, and create per-form sections, allowing a custom translation of displayed captions or hints.
This creation step needs a compilation of the executable with the {\f1\fs20 EXTRACTALLRESOURCES} conditional defined, {\i globally} to the whole application (a full {\i rebuild} is necessary after having added or suppressed this conditional from the {\i Project / Options / Folders-Conditionals} IDE field).
Then the {\f1\fs20 ExtractAllResources} global procedure is to be called somewhere in the code.
For instance, here is how this is implemented in @!TMainForm.FormShow!Lib\SQLite3\Samples\MainDemo\FileMain.pas@, for the framework main demo:
!procedure TMainForm.FormShow(Sender: TObject);
!begin
!!{$ifdef EXTRACTALLRESOURCES}
!!  ExtractAllResources(
!    // first, all enumerations to be translated
!!    [TypeInfo(TFileEvent),TypeInfo(TFileAction),TypeInfo(TPreviewAction)],
!    // then some class instances (including the TSQLModel will handle all TSQLRecord)
!!    [Client.Model],
!    // some custom classes or captions
!    [],[]);
!!  Close;
!{$else}
!  //i18nLanguageToRegistry(lngFrench);
!{$endif}
!  Ribbon.ToolBar.ActivePageIndex := 1;
!end;
A global {\f1\fs20 EXTRACTALLRESOURCES} conditional can be defined temporarly for the project: from the {\i Delphi} IDE, {\i Project/Options} then enabling the conditional, {\i Project/Run} to create the {\f1\fs20 .messages} file as expected, and finally {\i Project/Options} to undefine the {\f1\fs20 EXTRACTALLRESOURCES} conditional and rebuild a regular executable.
The {\f1\fs20 TFileEvent} and {\f1\fs20 TFileAction} enumerations RTTI information is supplied, together with the current {\f1\fs20 TSQLModel} instance. All {\f1\fs20 TSQLRecord} classes (and therefore properties) will be scanned, and all needed English caption text will be extracted.
The {\f1\fs20 Close} method is then called, since we don't want to use the application itself, but only extract all resources from the executable.
Running once the executable will create a {\f1\fs20 SynFile.messages} text file in the {\f1\fs20 SynFile.exe} folder, containing all English text:
$[TEditForm]
$Name.EditLabel.Caption=_2817614158   Name
$KeyWords.EditLabel.Caption=_3731019706   KeyWords
$
$[TLoginForm]
$Label1.Caption=_1741937413   &User name:
$Label2.Caption=_4235002365   &Password:
$
$[TMainForm]
$Caption=_16479868    Synopse mORMot demo - SynFile
$
$[Messages]
$2784453965=Memo
$2751226180=Data
$744738530=Safe memo
$895337940=Safe data
$2817614158=Name
$1741937413=&User name:
$4235002365=&Password:
$16479868= Synopse mORMot demo - SynFile
$940170664=Content
$3153227598=None
$3708724895=Page %d / %d
$2767358349=Size: %s
$4281038646=Content Type: %s
$2584741026=This memo is password protected.|Please click on the "Edit" button to show its content.
$3011148197=Please click on the "Extract" button to get its content.
$388288630=Signed,By %s on %s
$ (...)
The main section of this text file is named {\f1\fs20 [Messages]}. In fact, it contains all English extracted texts, as {\f1\fs20 NumericalKey=EnglishText} pairs. Note this will reflect the exact content of {\f1\fs20 resourcestring} or RTTI captions, including formating characters (like {\f1\fs20 %d}), and replacing line feeds ({\f1\fs20 #13}) by the special {\f1\fs20 |} character (a line feed is not expected on a one-line-per-pair file layout). Some other text lines are separated by a comma. This is usual for instance for hint values, as expected by the code.
As requested, each application form has its own section (e.g. {\f1\fs20 [TEditForm]}, {\f1\fs20 [TMainForm]}), proposing some default translation, specified by a numerical key (for instance {\f1\fs20 Label1.Caption} will use the text identified by 1741937413 in the {\f1\fs20 [Messages]} section). The underline character before the numerical key is used to refers to this value. Note that if no {\f1\fs20 _NumericalKey} is specified, a plain text can be specified, in order to reflect a specific use of the generic text on the screen.
:  Adding a new language
In order to translate the whole application into French, the following {\f1\fs20 FR.msg} file could be made available in the {\f1\fs20 SynFile.exe} folder:
$[Messages]
$2784453965=Texte
$2751226180=Données
$744738530=Texte sécurisé
$895337940=Données sécurisées
$2817614158=Nom
$1741937413=&Nom utilisateur:
$4235002365=&Mot de passe:
$16479868= Synopse mORMot Framework demo - SynFile
$940170664=Contenu
$3153227598=Vide
$3708724895=Page %d / %d
$2767358349=Taille: %s
$4281038646=Type de contenu: %s
$84741026=Le contenu de ce memo est protégé par un mot de passe.|Choisissez "Editer" pour le visualiser.
$3011148197=Choisissez "Extraire" pour enregistrer le contenu.
$388288630=Signé,Par %s le %s
$ (....)
Since no form-level custom captions (e.g. {\f1\fs20 [TLoginForm]}) have been defined in this {\f1\fs20 FR.msg} file, the default numerical values will be used. In our case, {\f1\fs20 Name.EditLabel.Caption} will be displayed using the text specified by 2817614158, i.e. {\f1\fs20 'Nom'}. You can specify a custom translation for a given field on any form: sometimes, the text should be adapted with a given context.
Note that the special characters {\f1\fs20 %s %d , |} markup was preserved: only the plain English text has been translated to the corresponding French.
:  Language selection
User Interface language can be specified at execution.
There are two ways to change the application language:
- Manual translation of every form;
- Hook of the common {\f1\fs20 TForm / TFrame} classes, for automatic translation.
In manual translation mode:
- You can change languages on the fly, i.e. no need to restart the application;
- But you must modify your code to explicitly translate the forms after their creation;
- And you won't be able to translate dialogs without sources (e.g. third-party dialogs).
{\f1\fs20 TForm/TFrame} hook, on its side, has the following behavior:
- You do not need to modify your code, since it will be global to the application;
- It will work also for any third-party dialog, even if you do not have the source of it;
- But you can't change the language on the fly: you need to restart the application.
:  Manual translation
Once for the application, you should call {\f1\fs20 SetCurrentLanguage()} to set the global {\f1\fs20 Language} object and all related {\i Delphi} locale settings.
The, in each {\f1\fs20 OnShow} event of any form, you should call {\f1\fs20 FormTranslateOne()} e.g.
!procedure TMyForm.FormShow(Sender: TObject);
!begin
!  Language.FormTranslateOne(self);
!end;
Another possibility may be to translate all already allocated forms at once, e.g. in the {\f1\fs20 OnShow} event of the application's main form:
! Language.FormTranslate([MainForm, FormTwo, FormAbout]);
Note that a list of already translated forms is maintained by the unit, when you call {\f1\fs20 FormTranslate()}.\line Therefore:
- All specified forms will be translated again by any further {\f1\fs20 SetCurrentLanguage()} call;
- But none of these forms must be freed after a {\f1\fs20 FormTranslate([])} call - use {\f1\fs20 FormTranslateOne()} instead to translate a given form once, e.g. for all temporary created forms.
:  TForm / TFrame hook
If the {\f1\fs20 USEFORMCREATEHOOK} conditional is defined, the {\f1\fs20 mORMoti18n.pas} unit will hook {\f1\fs20 TCustomForm.OnCreate} method to translate all its nested components. It will also intercept {\f1\fs20 TCustomFrame.Create()} to allow automatic translation of its content.
Since the language must be known at program startup, before any {\f1\fs20 TForm} is actually created, the language will be set in the Operating System registry.\line The {\f1\fs20 HKEY_CURRENT_USER\\Software\\[CompanyName]i18n\\} key should contain one value per application (i.e. the lowercase .exe file name without its path), which will identify the abbreviation of the expected language. If there is no entry in this registration key for the given application, the current Windows local will be used.
For instance, if you define {\f1\fs20 USEFORMCREATEHOOK} conditional for your project, and run at least e.g. once in @!TMainForm.FormShow!Lib\SQLite3\Samples\MainDemo\FileMain.pas@, for the framework main demo:
!  i18nLanguageToRegistry(lngFrench);
.. then it will set the main application language as French. At next startup, the unit will search for a {\f1\fs20 FR.msg} file, which will be used to translate all screen layout, including all RTTI-generated captions.
Of course, for a final application, you'll need to change the language by a common setting. See {\f1\fs20 i18nAddLanguageItems, i18nAddLanguageMenu} and {\f1\fs20 i18nAddLanguageCombo} functions and procedures to create your own language selection dialog, using a menu or a combo box, for instance.
:  Localization
Take a look at the {\f1\fs20 TLanguageFile} class. After the main language has been set, you can use the global {\f1\fs20 Language} instance in order to localize your application layout.
The {\f1\fs20 mORMoti18n} unit will register itself to some methods of {\f1\fs20 mORMot.pas}, in order to translate the RTTI-level text into the current selected language. See for instance {\f1\fs20 i18nDateText}.

[SAD-Ajax]
SourcePath=
IncludePath=
SourceFile=
Version=1.18
DisplayName=Ajax clients for mORMot

:Smart Mobile Studio
Did you hear from the great {\i @**Smart@} project?
It is an IDE and some source runtime able to develop and compile an Object-Pascal project into a {\i @*HTML 5@ / @*CSS 3@ / @*JavaScript@} embedded application. It does target AJAX Mobile application (i.e. {\i @*Android@} and {\i @*iPhone@/iPad} apps running {\i Web-Kit}) creation. You'll get an unique {\f1\fs20 .html} file containing the whole client-side application: it won't need any server side implementation. Using a third-party tool like {\i @*PhoneGap@}, you'd be able to supply your customers with true native applications, running without any network, and accessing the full power of any modern Smart Phone.
{\i Smart} is a great candidate for implementing rich client-side AJAX applications, to work with our client-server {\i mORMot} framework.
: Introduction to Smart coding
In order to interface {\i Smart} code with {\i mORMot}, we started implementing some low-level code to work with our @*REST@ful authentication scheme.
So we'll need to implement some Smart dedicated Open Source code implementing {\i crc32} and {\i SHA256} hashing.
:  First steps
It is very easy to work with Smart.
Open the IDE, create a new project, select for instance a "Console" template, then code your object pascal units, just as usual.
The only missing feature is a debugger integrated into the IDE. You'll have to debug your code from the {\i JavaScript} side (using the debugging tools featured in {\i Chrome}, for instance).
You have at hand the full power and readability of the great object pascal implementation of {\i Delphi Web Script}. See @http://code.google.com/p/dwscript about this great Open Source project.
No more type-less {\i JavaScript} coding, no more pseudo classes, no more endless nested delegates... Readable code, modern object oriented language (including strong typing, interfaces and inheritance), with a lot of existing and proven code base from a happy-sharing community. Just plain Object Pascal code, with the full power of the HTML 5 platform at hand.
It does not need external JavaScript library (like {\i jQuery} or such), but you can use any of those libraries, if needed.
:  Conversion rules
The {\i Delphi Web Script} implementation is very close to the {\i Delphi} / @*FPC@ implementation. It features a very modern object pascal syntax. But it will run with {\i JavaScript} as its runtime engine: you can see {\i JavaScript} as the replacement of {\i assembler} code for a regular {\i Delphi}/FPC compiler - and you can in fact write directly {\i JavaScript} code in the middle of a {\i Smart} unit using the {\f1\fs20 asm .. end} keywords.
As a consequence, when implementing low-level algorithms like {\i crc32} or {\i SHA256} with Smart, some genuineness is to be taken in account:
- There is no low level binary types like {\f1\fs20 byte, integer, cardinal, Int64, UInt64} which are all mapped as one {\f1\fs20 integer} type (since hashing use binary representation of the data, we must take care of this);
- You can't play with memory buffers directly in the language (this is a managed code, without any pointer nor memory allocation) - so we'll use {\f1\fs20 string} to handle memory buffers;
- You have to handle all code endianess by hand (by definition, {\i JavaScript} is endian-agnostic);
- There is no {\f1\fs20 char} type, just {\f1\fs20 string}.
But most high-level code could be shared between a {\i Delphi} application and a {\i Smart} application. For instance, when used within our {\i mORMot} framework, you may share the @*ORM@ definitions (via {\f1\fs20 class}) or the @*SOA@ contract definitions (via {\f1\fs20 @*interface@}), and most of your business logic.
:  CRC32 computation
So let's start with {\i @*crc32@} algorithm.
Here is the main computation code:
!var
!  crc32Tab: array [0..255] of integer;
!
!function crc32(aCRC32: integer; const data: string): integer;
!var i: integer;
!begin
!  result := (not aCRC32) shr 0;
!  for i := 1 to length(data) do
!    result := crc32Tab[(result xor ord(data[i])) and $ff] xor (result shr 8);
!  result := (not result) shr 0;
!end;
This is a standard implementation pattern, except for three remarks:
- We added {\f1\fs20 ... shr 0} in order to ensure that an {\f1\fs20 integer} variable will be maintained as an {\f1\fs20 UInt32} variable. Since {\i crc32} is a 32-bit hashing algorithm, we need to ensure that we'll only have positive values;
- No {\f1\fs20 byte} type is available here: so we'll explicitly call {\f1\fs20 ... and $ff} in order to truncate the {\f1\fs20 integer} value into its 8 bit content;
- Since the {\f1\fs20 string} type is used for data manipulation, we use {\f1\fs20 ord(data[i])} to retrieve each {\f1\fs20 byte} (or {\f1\fs20 char}) of the supplied text.
This implementation of the {\i crc32} algorithm expect a pre-computed table to be available. All {\i JavaScript} implementation of this algorithm (at least, all that I was able to found on Internet) use a fixed constant array. Since we are not afraid to write code any more in our AJAX application, and since it may help saving bandwidth and application size, we'll compute our own {\f1\fs20 crc32Tab[]} array content with the following code:
!procedure InitCrc32Tab;
!var i,n,crc: integer;
!begin // this code generates a 1KB table
!  for i := 0 to 255 do begin
!    crc := i;
!    for n := 1 to 8 do
!      if (crc and 1)<>0 then
!        // $edb88320 from polynomial p=(0,1,2,4,5,7,8,10,11,12,16,22,23,26)
!        crc := ((crc shr 1) xor $edb88320) shr 0 else
!        crc := crc shr 1;
!    CRC32Tab[i] := crc;
!  end;
!end;
Then we are able to use this code as such, for instance in a {\i Smart} console application:
!procedure TApplication.PopulateConsole;
!var i: integer;
!begin
!  InitCrc32Tab;
!  console.Writeln(IntToHex(crc32(0,'TestCRC32'),8));
!end;
The {\f1\fs20 InitCrc32Tab} shall be called only once, at application startup. Its execution is immediate. It won't delay your application display.
:  SHA256
The well-known @*SHA256@ algorithm is a proven way of creating an unique identifier from any data input. You can use it for instance to sign any content, or store efficiently a password. It is mathematically proven to be impossible to find out the input data from its hashed reduction (at least for the next decade of computer power). And it has a very low potential of "collision" (i.e. two diverse data having the same resulting hash). It is "Top Secret" enabled - U.S. National Institute of Standards and Technology says, "Federal agencies must use the SHA-2 family of hash functions for applications  that require collision resistance after 2010". This is the hashing pattern used within {\i mORMot}.
But it is also more complex than the {\i @*crc32@} algorithm. See @http://en.wikipedia.org/wiki/SHA-2
You have an optimized implementation in the {\f1\fs20 SynCrypto.pas} unit, with tuned {\i x86} assembler code, and provided regression tests. We'll implement a pure Object Pascal version, compatible with the {\i Smart / Delphi Web Script (DWS)} compiler.
First of all, we'll define a {\f1\fs20 record} type. We may have used a {\f1\fs20 class}, but since we have an extended {\f1\fs20 record} type at hand with {\i DWS} (including properties and methods), we will stay to it.
!type
!  TSHA256Buffer = array[0..63] of integer;
!  TSHAHash  = record
!    A,B,C,D,E,F,G,H: integer;
!  end;
!  TSHA256 = record
!  private
!    // Working hash
!    Hash: TSHAHash;
!    // 64bit msg length
!    MLen: integer;
!    // Block buffer
!    Buffer: TSHA256Buffer;
!    // Index in buffer
!    Index: integer;
!    // used by Update and Finalize
!    procedure Compress;
!  public
!    /// initialize SHA256 context for hashing
!    procedure Init;
!    /// update the SHA256 context with some data
!    procedure Update(const Data: string);
!    /// finalize and compute the resulting SHA256 hash Digest of all data
!    // affected to Update() method
!    // - returns the data as Hexadecimal
!    function Finalize: string;
!    /// compute SHA256 hexa digest of a given text
!    class function Compute(Data: string): string;
!  end;
The main {\f1\fs20 class function} can be used as such:
!  console.WriteLn(TSHA256.Compute('abc'));
And it will be implemented as:
!class function TSHA256.Compute(Data: string): string;
!var SHA: TSHA256;
!begin
!  SHA.Init;
!  SHA.Update(Data);
!  result := SHA.Finalize;
!end;
The initialization will be done with this method:
!procedure TSHA256.Init;
!begin
!  Hash.A := $6a09e667;
!  Hash.B := $bb67ae85;
!  Hash.C := $3c6ef372;
!  Hash.D := $a54ff53a;
!  Hash.E := $510e527f;
!  Hash.F := $9b05688c;
!  Hash.G := $1f83d9ab;
!  Hash.H := $5be0cd19;
!end;
Note that the {\i DWS} compiler will initialize all record content to zero by default. The corresponding {\i JavaScript} code will be emitted. So only {\f1\fs20 Hash.?} values are to be set explicitly: {\f1\fs20 Index} and {\f1\fs20 MLen} properties are already set to 0.
Then the following method will update the current hash with some supplied data:
!procedure TSHA256.Update(const Data: string);
!var Len, aLen, i: integer;
!    DataNdx: integer = 1;
!begin
!  Len := length(Data);
!  inc(MLen,Len shl 3);
!  while Len>0 do begin
!    aLen := 64-Index;
!    if aLen<=Len then begin
!      for i := 0 to aLen-1 do
!        Buffer[Index+i] := ord(Data[DataNdx+i]) and $ff;
!      dec(Len,aLen);
!      inc(DataNdx,aLen);
!!      Compress;
!      Index := 0;
!    end else begin
!      for i := 0 to Len-1 do
!        Buffer[Index+i] := ord(Data[DataNdx+i]) and $ff;
!      inc(Index,Len);
!      break;
!    end;
!  end;
!end;
The internal {\f1\fs20 Buffer[]}, which is expected to contain up to 64 bytes, is filled with the provided data, then the global {\f1\fs20 MLen} is refreshed, and the {\f1\fs20 Compress} method will do the proper hashing, when the 64 bytes buffer is full. The {\f1\fs20 Index} variable is used to track the number of bytes available in the internal {\f1\fs20 Buffer[]} array.
The {\f1\fs20 Finalize} method will compute the latest block, including the global length to the incoming data (with padding if needed), then will return the data as a hexadecimal string:
!function TSHA256.Finalize: string;
!var i: integer;
!begin
!  // Message padding
!  // 1. append bit '1' after Buffer
!  Buffer[Index]:= $80;
!  for i := Index+1 to 63 do
!    Buffer[i] := 0;
!  // 2. Compress if more than 448 bits, (no room for 64-bit length)
!  if Index>=56 then begin
!    Compress;
!    for i := 0 to 59 do
!      Buffer[i] := 0;
!  end;
!  // Write 64-bit Buffer length into the last bits of the last block
!  // (in big endian format) and do a final compress
!  Buffer[60] := (MLen and $ff000000)shr 24;
!  Buffer[61] := (MLen and $ff0000)shr 16;
!  Buffer[62] := (MLen and $ff00)shr 8;
!  Buffer[63] := MLen and $ff;
!  Compress;
!  // Hash -> Digest to big endian format
!  result := LowerCase(IntToHex(Hash.A,8)+IntToHex(Hash.B,8)+IntToHex(Hash.C,8)+
!    IntToHex(Hash.D,8)+IntToHex(Hash.E,8)+IntToHex(Hash.F,8)+IntToHex(Hash.G,8));
!  // Clear Data
!  Init;
!end;
Some endianess tricks are used in the above code. But it remains easy to follow and maintain.
The main hash computation is performed in the {\f1\fs20 Compress} method, as such:
!const
!  K: TSHA256Buffer = ([
!   $428a2f98, $71374491, $b5c0fbcf, $e9b5dba5, $3956c25b, $59f111f1,
!   $923f82a4, $ab1c5ed5, $d807aa98, $12835b01, $243185be, $550c7dc3,
!   $72be5d74, $80deb1fe, $9bdc06a7, $c19bf174, $e49b69c1, $efbe4786,
!   $0fc19dc6, $240ca1cc, $2de92c6f, $4a7484aa, $5cb0a9dc, $76f988da,
!   $983e5152, $a831c66d, $b00327c8, $bf597fc7, $c6e00bf3, $d5a79147,
!   $06ca6351, $14292967, $27b70a85, $2e1b2138, $4d2c6dfc, $53380d13,
!   $650a7354, $766a0abb, $81c2c92e, $92722c85, $a2bfe8a1, $a81a664b,
!   $c24b8b70, $c76c51a3, $d192e819, $d6990624, $f40e3585, $106aa070,
!   $19a4c116, $1e376c08, $2748774c, $34b0bcb5, $391c0cb3, $4ed8aa4a,
!   $5b9cca4f, $682e6ff3, $748f82ee, $78a5636f, $84c87814, $8cc70208,
!   $90befffa, $a4506ceb, $bef9a3f7, $c67178f2]);
!
!procedure TSHA256.Compress;
!var W: TSHA256Buffer;
!    H: TSHAHash = Hash;
!    i, t1, t2: integer;
!begin
!  for i := 0 to 15 do
!    W[i]:= (((Buffer[i*4] shl 24)shr 0)or(Buffer[i*4+1] shl 16)or
!           (Buffer[i*4+2] shl 8)or Buffer[i*4+3]) shr 0;
!  for i := 16 to 63 do
!    W[i] := ((((W[i-2]shr 17)or(W[i-2]shl 15))xor((W[i-2]shr 19)or(W[i-2]shl 13))
!      xor (W[i-2]shr 10))+W[i-7]+(((W[i-15]shr 7)or(W[i-15]shl 25))
!      xor ((W[i-15]shr 18)or(W[i-15]shl 14))xor(W[i-15]shr 3))+W[i-16])shr 0;
!  for i := 0 to high(W) do begin
!    t1 := (H.H+(((H.E shr 6)or(H.E shl 26))xor((H.E shr 11)or(H.E shl 21))xor
!      ((H.E shr 25)or(H.E shl 7)))+((H.E and H.F)xor(not H.E and H.G))+K[i]+W[i])shr 0;
!    t2 := ((((H.A shr 2)or(H.A shl 30))xor((H.A shr 13)or(H.A shl 19))xor
!      ((H.A shr 22)xor(H.A shl 10)))+((H.A and H.B)xor(H.A and H.C)xor(H.B and H.C))) shr 0;
!    H.H := H.G; H.G := H.F; H.F := H.E; H.E := (H.D+t1)shr 0;
!    H.D := H.C; H.C := H.B; H.B := H.A; H.A := (t1+t2)shr 0;
!  end;
!  Hash.A := (Hash.A+H.A)shr 0;
!  Hash.B := (Hash.B+H.B)shr 0;
!  Hash.C := (Hash.C+H.C)shr 0;
!  Hash.D := (Hash.D+H.D)shr 0;
!  Hash.E := (Hash.E+H.E)shr 0;
!  Hash.F := (Hash.F+H.F)shr 0;
!  Hash.G := (Hash.G+H.G)shr 0;
!  Hash.H := (Hash.H+H.H)shr 0;
!end;
A {\f1\fs20 K: TSHA256Buffer} constant table is used. Note the non standard definition of {\i DWS} for a {\f1\fs20 const array}: it will use {\f1\fs20 .. = ([...]);} instead of {\f1\fs20 .. = ();} as in classical {\i Object Pascal}.
The hash is computed from an internal {\f1\fs20 W[]} array, which is filled with the binary representation of the supplied {\f1\fs20 Buffer[]} bytes. That is, {\f1\fs20 Buffer: array[0..63] of byte} is first un-serialized in {\f1\fs20 W: array[0..15] of cardinal}.
Then the SHA-256 algorithm is performed in its most simple rolled version. An un-rolled version is not mandatory here, in our managed {\i JavaScript} runtime environment.
The only non obvious part of the above code is the use of {\f1\fs20 ... shr 0} to enforce only positive 32-bit integers (aka {\f1\fs20 cardinal}) are used during the computation.

To illustrate that, here is some interesting question published in our forum - see @https://synopse.info/forum/viewtopic.php?id=1011 - from a customer:
{\i I have written a test server (using {\f1\fs20 TSQLRestServerDB}), which has a service that provides basic access to the database as follows:}
!  TDatabaseRecord=record
!    Name: string;
!    Age: integer;
!  end;
!  TDatabaseRecordArray=array of TDatabaseRecord;
!
!  IDatabaseService=interface(IInvokable)
!    ['{9ABC8235-5102-4C02-8469-0BCA606C1CF0}']
!    function Add(const ARecord: TDatabaseRecord): integer;
!    procedure Delete(AIndex: integer);
!    procedure List(var ARecords: TDatabaseRecordArray);
!  end;
{\i The class implementing this service uses a class inherited from {\f1\fs20 TSQLRecord} to access the database. This class has similar parameters to the record {\f1\fs20 TDatabaseRecord}.}
!TSQLPerson = class(TSQLRecord)
!  private
!    FName: RawUTF8;
!    FAge: integer;
!  published
!    property Age: integer read FAge write FAge;
!    property Name: RawUTF8 read fName write fName;
!  end;
{\i The client uses the interface to connect to the server and everything works OK.}
{\i But then I thought, why don't use the {\f1\fs20 TSQLPerson} class directly in the client, so it can access directly to the database bypassing all the process the service has to do? It worked also, of course.}
{\i So my question is: which way is better? Are the services meant to be used like this? Is it safer to access directly to the database from clients? I started with the services way thinking that may be more secure the usage of services instead of direct accessing. But since the framework already has a lot of security functionalities maybe I am overloading the server without achieving anything...}
{\i Note that my clients will be AJAX basically, not Delphi clients (don't know if it is relevant).}
In fact, for {\f1\fs20 TSQLRecord} / @*ORM@ remote access, you have already all @*Client-Server@ @*CRUD@ operations available. It has been optimized a lot (e.g. with a cache and other nice features), so I do not think reinventing a CRUD / database service is worth the prize. You have secure access to the ORM classes, with user/group attributes. Almost everything is created by code, just from the {\f1\fs20 TSQLRecord} class definition, via @*RTTI@. So it may be faster (and safer) to rely on the existing.
If you use AJAX clients, high-level only services, with records (which will be mapped as JavaScript objects) do make sense. Our ORM / {\f1\fs20 TSQLRecord} / {\f1\fs20 TSQLRest} classes do amazing work in the world of {\i Delphi} classes (including filtering or validation, strong typing, shared model between client and server, and abstract {\f1\fs20 TSQLRest} CRUD methods), but for AJAX clients, the benefit is less obvious. Note that in order to serialize the {\f1\fs20 TDatabaseRecord} records as pure JSON (i.e. AJAX-friendly), you will need to code the writer and reader routines by hand - see @51@ - whereas it is already included with {\f1\fs20 TSQLPerson}. So AJAX does make a difference here. That's why, in case of AJAX clients, I will tend to make services with small records dedicated to your client applications, or even {\f1\fs20 TSQLRecord} (which will be published directly as JavaScript objects) in some cases. It will depend on the nature of your business logic.
Architecturally speaking, the question faces another point: the layer separation. It will now be time to enter into DDD material. See @68@.

[SDD]
Owner=SRS
Order=SRS
; Owner: [SDD-*] -> * reference; Order=SRS -> [SDD-*] have no sub items
Name=Software Design Document
Purpose=Summarize the software DI implementation for QA review
PreparedBy=Arnaud Bouchez
ReviewedBy=
ApprovedBy=
Revision=1.18
RevisionDate=
RevisionDescription=Initial Version
; Revision* multiple revision Table: ignored values are taken from current, older below
; [SDD-*] sections contain details for each SRS, [SDD-SER-03] or [SDD-DI-4.10.6] e.g.
; [SDD-*] are displayed as they appear in the [SRS-*] sections
DocumentFrontPage=ProjectDetails,Warning,PeopleDetails,RevisionDetails,AddPurpose
WriteTableOfContent=Yes
; Write global Table Of Contents at the end of the file
; Write global Table Of Contents at the end of the file
TableOfContentsAtTheBeginning=Yes
; if the Table of Contents must be at the beginning (default=No=at the end of the file)
DocumentIndex=Pictures,Source,Index

:Introduction
: Documentation overview
The whole Software documentation process follows the typical steps of this diagram:
%%FMEADI
: Purpose
This @SDD@ applies to the release of the {\i Synopse mORMot Framework} library.
It summarizes the software implementation of each design input as specified by the @DI@.
This document is divided into the main parts of the Software implementation:
\LayoutPage
Inside this sections, source code or User Interface modifications are detailed for every @SRS@ item.
: Responsibilities
- Synopse will try to correct any identified issue;
- The Open Source community will create tickets in a public Tracker web site located at @https://synopse.info/fossil ;
- Synopse work on the framework is distributed without any warranty, according to the chosen license terms;
- This documentation is released under the GPL (GNU General Public License) terms, without any warranty of any kind.
=[GPL]

[SDD-DI-2.1.1]
; SRS-DI-2.1.1 - The framework must be Client-Server oriented
:Implementation
The @*Client-Server@ aspect of the framework is implemented in the @!TSQLRecord,TSQLRest,TSQLRestServer,TSQLRestClientURI,TSQLTableJSON!Lib\SQLite3\mORMot.pas@ unit, with the {\f1\fs20 TSQLRestServer} and {\f1\fs20 TSQLRestClientURI} classes.
Both classes inherit from a generic {\f1\fs20 TSQLRest} class, which implements some generic database access methods and properties (through @*ORM@ model for objects descending from {\f1\fs20 TSQLRecord} or table-based query using {\f1\fs20 TSQLTableJSON}).

[SDD-DI-2.1.1.1]
; SRS-DI-2.1.1.1 - A RESTful mechanism must be implemented
:Implementation
The @*REST@ful mechanism is implemented using the {\f1\fs20 URI} method of both {\f1\fs20 TSQLRestServer} and {\f1\fs20 TSQLRestClientURI} classes, as defined in the @!TSQLRest,TSQLRestServer,TSQLRestClientURI!Lib\SQLite3\mORMot.pas@ unit.
: Server-Side
In the {\f1\fs20 TSQLRestServer} class, the {\f1\fs20 URI} method is defined as {\f1\fs20 public}, and must implement the actual database query or update, according to the REST request:
!procedure TSQLRestServer.URI(var Call: TSQLRestServerURIParams);
The purpose of this method is to:
- Return internal database state count (used for caching);
- Retrieve URI expecting the RESTful {\f1\fs20 'ModelRoot[/TableName[/TableID[/BlobFieldName]]]'} format;
- Call appropriate database commands, by using the protected {\f1\fs20 EngineList EngineRetrieve EngineAdd EngineUpdate EngineDelete EngineRetrieveBlob EngineUpdateBlob} methods.
The {\f1\fs20 TSQLRestServer} class itself doesn't implement these database command methods: they are all defined as {\f1\fs20 virtual; abstract;}. Children classes must override these virtual methods, and implement them using the corresponding database engine.
: Client-Side
In the {\f1\fs20 TSQLRestClientURI} class, the {\f1\fs20 URI} method is defined as {\f1\fs20 protected} and as {\f1\fs20 virtual; abstract;}. Children classes must override this method, and implement the remote database query or update, according to the REST request, and its internal protocol.

[SDD-DI-2.1.1.2]
; SRS-DI-2.1.1.2 - Commmunication should be available directly in the same process memory, or remotly using Named Pipes, Windows messages or HTTP/1.1 protocols

[SDD-DI-2.1.1.2.1]
; SRS-DI-2.1.1.2.1 - Client-Server Direct communication inside the same process
:Implementation
The in-process communication is implemented by using a global function, named {\f1\fs20 URIRequest} and defined in @!TSQLRestServer,URIRequest,USEFASTMM4ALLOC,TSQLRestClientURIDll.Create!Lib\SQLite3\mORMot.pas@:
!function URIRequest(url, method, SendData: PUTF8Char; Resp, Head: PPUTF8Char): Int64Rec; cdecl;
: Server-Side
This function can be exported from a DLL to remotely access to a {\f1\fs20 TSQLRestServer}, or used in the same process:
- Use {\f1\fs20 TSQLRestServer.ExportServer} to assign a server to this function;
- Return {\i 501 NOT IMPLEMENTED} error if no {\f1\fs20 TSQLRestServer.ExportServer} has been assigned yet;
- Memory for {\f1\fs20 Resp} and {\f1\fs20 Head} parameters are allocated with {\f1\fs20 GlobalAlloc()} Windows API function: client must release this pointers with {\f1\fs20 GlobalFree()} after having retrieved their content - you can force using the {\i Delphi} heap (and {\f1\fs20 GetMem} function which is much faster than {\f1\fs20 GlobalAlloc}) by setting the {\f1\fs20 USEFASTMM4ALLOC} variable to TRUE: in this case, client must release this pointers with {\f1\fs20 Freemem()}.
: Client-Side
The Client should simply use a {\f1\fs20 TSQLRestClientURIDll} instance to access to an exported {\f1\fs20 URIRequest()} function.

[SDD-DI-2.1.1.2.2]
; SRS-DI-2.1.1.2.2 - Client-Server Named Pipe communication
:Implementation
: Server-Side
The communication is implemented by using the {\f1\fs20 TSQLRestServer} class, defined in @!TSQLRestServer.ExportServerNamedPipe,TSQLRestClientURINamedPipe.Create,TSQLRestClientURI!Lib\SQLite3\mORMot.pas@.
This class implements a server over Named Pipe communication, when its {\f1\fs20 ExportServerNamedPipe} method is called.
: Client-Side
A dedicated {\f1\fs20 TSQLRestClientURINamedPipe} class has been defined. It inherits from {\f1\fs20 TSQLRestClientURI}, and override its {\f1\fs20 URI} protected method so that it communicates using a specified Named Pipe.

[SDD-DI-2.1.1.2.3]
; SRS-DI-2.1.1.2.3 - Client-Server Windows Messages communication
:Implementation
Communication using Windows Messages is very handy and efficient on the same computer. It's also perfectly safe, because, by design, it can't be access remotely. Performances for small messages is also excellent. Named pipe could be faster only when bigger messages are transmitted.
: Server-Side
The communication is implemented by using the {\f1\fs20 TSQLRestServer} class, defined in @!TSQLRestClientURI,TSQLRestServer.ExportServerMessage,TSQLRestClientURIMessage.Create!Lib\SQLite3\mORMot.pas@.
This class implements a server over Windows Messages communication, when its {\f1\fs20 ExportServerMessage} method is called.
: Client-Side
A dedicated {\f1\fs20 TSQLRestClientURIMessage} class has been defined. It inherits from {\f1\fs20 TSQLRestClientURI}, and override its {\f1\fs20 URI} protected method so that it communicates using Windows Messages.

[SDD-DI-2.1.1.2.4]
; SRS-DI-2.1.1.2.4 - Client-Server HTTP/1.1 protocol communication
:Implementation
: Server-Side
The communication is not implemented directly in the {\f1\fs20 TSQLRestServer} class, defined in @!TSQLRestClientURI,TSQLRestServer.URI!Lib\SQLite3\mORMot.pas@, but by a dedicated {\f1\fs20 TSQLHttpServer} class defined in @!TSQLHttpServer.Create,TSQLHttpServer.DBServer,TSQLHttpServer.AddServer!Lib\SQLite3\mORMotHttpServer.pas@.
This class will instantiate a {\f1\fs20 THttpServerGeneric} instance, defined in @!THttpServer.Create,THttpApiServer.Create,THttpServerGeneric.Request,THttpServerGeneric.OnRequest!Lib\SynCrtSock.pas@, which implements a HTTP/1.1 server over TCP/IP communication.
This server can be implemented by two means:
- Via {\f1\fs20 THttpApiServer} for using the fast kernel-mode {\i http.sys} server;
- Via {\f1\fs20 THttpServer}, which is an optimized pure {\i Delphi} HTTP/1.1 compliant server, using {\i Thread pool} to reduce resources, and provide best possible performance in user land.
You can register several {\f1\fs20 TSQLRestServer} instance to the same HTTP server, via its {\f1\fs20 AddServer} method.  Each {\f1\fs20 TSQLRestServer} class must have an unique {\f1\fs20 Model.Root} value, to identify which instance must handle a particular request from its URI root string.
A dedicated property, named {\f1\fs20 DBServer}, is an array to all registered {\f1\fs20 TSQLRestServer} instances, which are used to process any request, and answer to it by using the corresponding {\f1\fs20 URI} method - via the {\f1\fs20 OnRequest} standard event prototype.
: Client-Side
A dedicated {\f1\fs20 TSQLHttpClient} class has been defined in @!TSQLHttpClient.Create!Lib\SQLite3\mORMotHttpClient.pas@. It inherits from {\f1\fs20 TSQLRestClientURI}, and override its {\f1\fs20 URI} protected method so that it communicates using HTTP/1.1 protocol over TCP/IP, according to the supplied HTTP address name.
By default, {\f1\fs20 TSQLHttpClient} maps to a {\f1\fs20 TSQLHttpClientWinHTTP} class, which was found out to perform well on most configurations and networks (whereas {\f1\fs20 TSQLHttpClientWinSock} should be a bit faster on a local computer).

[SDD-DI-2.1.2]
; SRS-DI-2.1.2 - UTF-8 JSON format must be used to communicate
:Implementation
The JSON parsing and producing is implemented in the @!TTextWriter.Create,TTextWriter.AddJSONEscape,IsJSONString,JSONDecode,JSONEncode,JSONEncodeArray,GetJSONField,JSON_CONTENT_TYPE!Lib\SynCommons.pas@ and @!TSQLTable.GetJSONValues,TSQLTableJSON.Create,TSQLTableJSON.UpdateFrom,TJSONWriter.Create,TSQLRecord.CreateJSONWriter,TSQLRecord.GetJSONValues,GetJSONObjectAsSQL,UnJSONFirstField!Lib\SQLite3\mORMot.pas@ units.
The JSON encoding and decoding is handled at diverse levels:
- With some JSON-dedicated functions and classes;
- At the database record level;
- At the database request table level.
: JSON-dedicated functions and classes
The main class for producing JSON content is {\f1\fs20 TJSONWriter}. This class is a simple writer to a Stream, specialized for the JSON format. Since it makes
use of an internal buffer, and avoid most temporary {\f1\fs20 string} allocation ({\i e.g.} using the stack instead of a temporary {\f1\fs20 string} via {\f1\fs20 IntToStr()} when converting a numerical value to text), it is much faster than a string append (standard {\i Delphi} {\f1\fs20 string := string+string} clauses) to produce its content. In particular, its {\f1\fs20 AddJSONEscape} method will handle JSON content escape, according to the official JSON RFC - see @http://www.ietf.org/rfc/rfc4627.txt paragraph 2.5, directly into the destination buffer. It was also designed to scales well on multi-core sytems.
Some JSON-dedicated function are also available:
- {\f1\fs20 GetJSONObjectAsSQL} decodes a JSON fields object into an @*UTF-8@ encoded SQL-ready statement;
- {\f1\fs20 IsJSONString} returns TRUE if the supplied content must be encoded as a JSON string according to the JSON encoding schema, i.e. if it's some null/false/true content or any pure numerical data (integer or floating point);
- {\f1\fs20 UnJSONFirstField} can be used to retrieve the FIRST field value of the FIRST row, from a JSON content: it may be useful to get an ID without converting the whole JSON content into a {\f1\fs20 TSQLTableJSON};
- {\f1\fs20 JSONEncode} and {\f1\fs20 JSONDecode} functions are available to directly encode or decode some UTF-8 JSON content (used in the remote @*Service@ implementation, for instance).
: Database record level
The {\f1\fs20 TJSONWriter} class (based on {\f1\fs20 TTextWriter}) is used by the {\f1\fs20 GetJSONValues} method of the {\f1\fs20 TSQLRecord} class to get all the data of a database record as JSON content.
Here is an extract of the main loop of this method:
!procedure TSQLTable.GetJSONValues(JSON: TStream; Expand: boolean;
!  RowFirst: integer=0; RowLast: integer=0);
!  (...)
!    for R := RowFirst to RowLast do
!    begin
!      if Expand then
!        W.Add('{');
!      for F := 0 to FieldCount-1 do
!      begin
!        if Expand then
!          W.AddString(W.ColNames[F]); // '"'+ColNames[]+'":'
!        if Assigned(QueryTables) then
!        if IsJSONString(U^) then
!        begin
!          W.Add('"');
!          W.AddJSONEscape(U^,0);
!          W.Add('"');
!        end else
!          W.AddNoJSONEscape(U^,0);
!        W.Add(',');
!        inc(U); // points to next value
!      end;
!      W.CancelLastComma; // cancel last ','
!      if Expand then
!        W.Add('}');
!      W.Add(',');
!    end;
!  (...)
: Database request table level
Most high-level Client-sided list request methods returns a {\f1\fs20 TSQLTableJSON} instance as a result. This {\f1\fs20 TSQLTableJSON} class has been created from a pure JSON content, retrieved from the Server using on of the protocols defined in @SRS-DI-2.1.1.2@.
Its {\f1\fs20 Create} constructor method call its internal {\f1\fs20 protected} method named {\f1\fs20 FillFrom()}, which make the JSON conversion into pure @*UTF-8@ text fields, as expected by the {\f1\fs20 TSQLTable} class and its various {\f1\fs20 Get*()} methods. The {\f1\fs20 FillFrom()} method implements a very fast parsing of the supplied JSON content, then un-escape its content according to the JSON RFC quoted above.
: Fast JSON parsing
When it deals with parsing some (textual) content, two directions are usually envisaged. In the XML world, you have usually to make a choice between:
- A DOM parser, which creates an in-memory tree structure of objects mapping the XML nodes;
- A @*SAX@ parser, which reads the XML content, then call pre-defined {\i events} for each XML content element.
In fact, DOM parsers use internally a SAX parser to read the XML content. Therefore, with the overhead of object creation and their property initialization, DOM parsers are typically three to five times slower than SAX. But, DOM parsers are much more powerful for handling the data: as soon as it's mapped in native objects, code can access with no time to any given node, whereas a SAX-based access will have to read again the whole XML content.
Most JSON parser available in {\i Delphi} use a DOM-like approach. For instance, the {\i DBXJSON} unit included since {\i Delphi} 2010 or the {\i SuperObject} library create a class instance mapping each JSON node.
In a JSON-based Client-Server ORM like ours, profiling shows that a lot of time is spent in JSON parsing, on both Client and Server side. Therefore, we tried to optimize this part of the library.
In order to achieve best speed, we try to use a mixed approach:
- All the necessary conversion (e.g. un-escape text) is made in-memory, from and within the JSON buffer, to avoid memory allocation;
- The parser returns {\i pointers} to the converted elements (just like the {\i vtd-xml} library).
In practice, here is how it is implemented:
- A private copy of the source JSON data is made internally (so that the Client-Side method used to retrieve this data can safely free all allocated memory);
- The source JSON data is parsed, and replaced by the UTF-8 text un-escaped content, in the same internal buffer (for example, strings are un-escaped and #0 are added at the end of any field value; and numerical values remains text-encoded in place, and will be extracted into {\f1\fs20 Int64} or {\f1\fs20 @*double@} only if needed);
- Since data is replaced in-memory (JSON data is a bit more verbose than pure UTF-8 text so we have enough space), no memory allocation is performed during the parsing: the whole process is very fast, not noticeably slower than a SAX approach;
- This very profiled code (using pointers and tuned code) results in a very fast parsing and conversion.
This parsing "magic" is done in the {\f1\fs20 GetJSONField} function, as defined in the @!GetJSONField!Lib\SynCommons.pas@ unit:
!/// decode a JSON field in an UTF-8 encoded buffer (used in TSQLTableJSON.Create)
!// - this function decodes in the P^ buffer memory itself (no memory allocation
!// or copy), for faster process - so take care that it's an unique string
!// - PDest points to the next field to be decoded, or nil on any unexpected end
!// - null is decoded as nil
!// - '"strings"' are decoded as 'strings'
!// - strings are JSON unescaped (and \u0123 is converted to UTF-8 chars)
!// - any integer value is left as its ascii representation
!// - wasString is set to true if the JSON value was a "string"
!// - works for both field names or values (e.g. '"FieldName":' or 'Value,')
!// - EndOfObject (if not nil) is set to the JSON value char (',' ':' or '}' e.g.)
!function GetJSONField(P: PUTF8Char; out PDest: PUTF8Char;
!  wasString: PBoolean=nil; EndOfObject: PUTF8Char=nil): PUTF8Char;
This function allows to iterate throughout the whole JSON buffer content, retrieving values or property names, and checking {\f1\fs20 EndOfObject} returning value to handle the JSON structure.
This in-place parsing of textual content is one of the main reason why we used UTF-8 (via {\f1\fs20 RawUTF8}) as the common string type in our framework, and not the generic {\f1\fs20 string} type, which will have introduced a memory allocation and a char-set conversion.
For instance, here is how JSON content is converted into SQL, as fast as possible:
!function GetJSONObjectAsSQL(var P: PUTF8Char; const Fields: TRawUTF8DynArray;
!  Update, InlinedParams: boolean): RawUTF8;
! (...)
!    // get "COL1"="VAL1" pairs, stopping at '}' or ']'
!    FieldsCount := 0;
!    repeat
!!      FU := GetJSONField(P,P);
!      inc(Len,length(FU));
!      if P=nil then break;
!      Fields2[FieldsCount] := FU;
!!      Values[FieldsCount] := GetValue; // update EndOfObject
!      inc(FieldsCount);
!    until EndOfObject in [#0,'}',']'];
!    Return(@Fields2,@Values,InlinedParams);
!  (...)
And the sub-function {\f1\fs20 GetValue} makes use of {\f1\fs20 GetJSONField} also:
!function GetValue: RawUTF8;
!var wasString: boolean;
!    res: PUTF8Char;
!begin
!  res := P;
!  if (PInteger(res)^ and $DFDFDFDF=NULL_DF) and (res[4] in [#0,',','}',']'])  then
!    /// GetJSONField('null') returns '' -> check here to make a diff with '""'
!    result := 'null' else begin
!    // any JSON string or number or 'false'/'true' in P:
!!    res := GetJSONField(res,P,@wasString,@EndOfObject);
!    if wasString then
!      if not InlinedParams and
!         (PInteger(res)^ and $00ffffff=JSON_BASE64_MAGIC) then
!        // \\uFFF0base64encodedbinary -> 'X''hexaencodedbinary'''
!        // if not inlined, it can be used directly in INSERT/UPDATE statements
!        result := Base64MagicToBlob(res+3) else
!        { escape SQL strings, cf. the official SQLite3 documentation }
!        result := QuotedStr(pointer(res),'''') else
!      result := res;
!  end;
!  Inc(Len,length(result));
!end;
This code will create a string for each key/value in {\f1\fs20 Fields2[]} and {\f1\fs20 Values[]} arrays, but only once, with the definitive value (even single quote escape and BLOB un-serialize from @*Base64@ encoding are performed directly from the JSON buffer).

[SDD-DI-2.1.3]
; SRS-DI-2.1.3 - The framework must use an innovative ORM (Object-relational mapping) approach, based on classes RTTI (Runtime Type Information)
:Implementation
Some {\i Delphi} @*RTTI@ (Runtime Type Information) objects and classes are implemented in the @!TClassProp,TClassType,TEnumType,TTypeInfo,TSQLRecord.ClassProp,TSQLRecord.GetJSONValues,TPropInfo.GetValue,TPropInfo.SetValue,TSQLRecordProperties!Lib\SQLite3\mORMot.pas@ unit. The {\i Synopse mORMot Framework} uses this custom functions and objects in order to access to the {\i Delphi} @*RTTI@.
The generic functions supplied by the standard {\f1\fs20 TypInfo.pas} unit where not found to be easy to use: there are some record types from one hand, which details the internal @*RTTI@ memory layout generated by the compiler, and there are some functions on the other hand. So the framework unified both RTTI memory layout and methods by defining some {\f1\fs20 object} types (i.e. not {\i Delphi} classes, but raw objects which can map directly the RTTI memory layout via a {\f1\fs20 pointer}) with some methods dedicated for RTTI handling and @*ORM@. These {\f1\fs20 object} types are {\f1\fs20 TClassProp, TClassType, TEnumType, TTypeInfo} and {\f1\fs20 TPropInfo}.
Since this ORM is the core of the framework, the code of most of these objects has been tuned for performance: quit all of the methods have two versions in the framework, one in pure pascal code (easy to maintain and understand, and @*64-bit@ compatible), and one in optimized i386 assembler.
As a result, ORM code based on RTTI is fairly easy to use. See for example who a database field index is retrieved for a {\f1\fs20 TSQLRecord} class:
!function ClassFieldIndex(ClassType: TClass; const PropName: shortstring): integer;
!var P: PPropInfo;
!    CP: PClassProp;
!begin
!  if ClassType<>nil then
!  begin
!    CP := InternalClassProp(ClassType);
!    if CP<>nil then
!    begin
!      P := @CP^.PropList;
!      for result := 0 to CP^.PropCount-1 do
!        if IdemPropName(P^.Name,PropName) then
!          exit else
!          P := P^.Next;
!    end;
!  end;
!  result := -1;
!end;
Internally, the {\f1\fs20 TSQLRecord} will cache some of this RTTI derived data into an internal {\f1\fs20 TSQLRecordProperties} instance, global for the whole process. For instance, the method used to retrieve a field index from its property name is the following:
!function TSQLRecordProperties.FieldIndex(const PropName: shortstring): integer;
!begin
!  if self<>nil then
!  for result := 0 to high(Fields) do
!    if IdemPropName(Fields[result]^.Name,PropName) then
!      exit;
!  result := -1;
!end;
And will be available from {\f1\fs20 TSQLRecord.RecordProps.FieldIndex}.
:TSQLRecord table properties
: Per-class variable needed
For our ORM, we needed a {\i class variable} to be available for each {\f1\fs20 TSQLRecord} class type. This variable is used to store the properties of this class type, i.e. the database Table properties (e.g. table and column names and types) associated with a particular {\f1\fs20 TSQLRecord} class, from which all our ORM objects inherit.
The {\f1\fs20 class var} statement was not enough for us:
- It's not available on earlier {\i Delphi} versions, and we try to have our framework work with {\i Delphi} 6-7;
- This {\f1\fs20 class var} instance will be shared by all classes inheriting from the class where it is defined - and we need ONE instance PER class type, not ONE instance for ALL
We need to find another way to implement this {\i class variable}. An unused VMT slot in the class type description was identified, then each class definition was patched in the process memory to contain our class variable.
: Patching a running process code
The first feature we have to do is to allow on-the-fly change of the assembler code of a process.
When an executable is mapped in RAM, the memory page corresponding to the process code is marked as {\i Read Only}, in order to avoid any security attack from the outside. Only the current process can patch its own code.
We'll need to override a {\f1\fs20 pointer} value in the code memory. The following function, defined in {\f1\fs20 SynCommons.pas} will handle it:
!procedure PatchCodePtrUInt(Code: PPtrUInt; Value: PtrUInt);
!var RestoreProtection, Ignore: DWORD;
!begin
!  if VirtualProtect(Code, SizeOf(Code^), PAGE_EXECUTE_READWRITE, RestoreProtection) then
!  begin
!    Code^ := Value;
!    VirtualProtect(Code, SizeOf(Code^), RestoreProtection, Ignore);
!    FlushInstructionCache(GetCurrentProcess, Code, SizeOf(Code^));
!  end;
!end;
The {\f1\fs20 VirtualProtect} low-level Windows API is called to force the corresponding memory to be written (via the {\f1\fs20 PAGE_EXECUTE_READWRITE} flag), then modify the corresponding {\f1\fs20 pointer} value, then the original memory page protection setting (should be {\f1\fs20 PAGE_EXECUTE_READ}) is restored.
According to the MSDN documentation, we'd need to flush the CPU operation cache in order to force the modified code to be read on next access.
: Per-class variable in the VMT
The VMT is the {\i Virtual-Method Table}, i.e. a Table which defines every {\i Delphi} {\f1\fs20 class}. In fact, every {\i Delphi} {\f1\fs20 class} is defined internally by its VMT, contains a list of pointers to the {\f1\fs20 class}’s {\f1\fs20 virtual} methods. This VMT also contains non-method values, which are class-specific information at negative offsets:
|%30%10%60
|\b Name|Offset|Description\b0
|{\f1\fs20 vmtSelfPtr}|–76|points back to the beginning of the table
|{\f1\fs20 vmtIntfTable}|–72|{\f1\fs20 TObject.GetInterfaceTable} method value
|{\f1\fs20 vmtAutoTable}|–68|class’s automation table (deprecated)
|{\f1\fs20 vmtInitTable}|–64|reference-counted fields type information
|{\f1\fs20 vmtTypeInfo}|–60|the associated RTTI type information
|{\f1\fs20 vmtFieldTable}|–56|field addresses
|{\f1\fs20 vmtMethodTable}|–52|method names
|{\f1\fs20 vmtDynamicTable}|–48|{\f1\fs20 dynamic} methods table
|{\f1\fs20 vmtClassName}|–44|{\f1\fs20 PShortString} of the class name
|{\f1\fs20 vmtInstanceSize}|–40|bytes needed by one class Instance
|{\f1\fs20 vmtParent}|–36|parent VMT
|%
We'll implement the low-level trick as detailed in this reference article available at @http://hallvards.blogspot.com/2007/05/hack17-virtual-class-variables-part-ii.html in order to use the {\f1\fs20 vmtAutoTable} deprecated entry in the VMT. This entry was used in {\i Delphi} 2 only for implementing {\i Automation}. Later version of {\i Delphi} (our goal) won't use it any more. But the slot is still here, ready for being used by the framework.
We'll therefore be able to store a pointer to the {\f1\fs20 TSQLRecordProperties} instance corresponding to a {\f1\fs20 TSQLRecord} class, which will be retrieved as such:
!class function TSQLRecord.RecordProps: TSQLRecordProperties;
!begin
!  if Self<>nil then begin
!    result := PPointer(PtrInt(Self)+vmtAutoTable)^;
!    if result=nil then
!      result := PropsCreate(self);
!  end else
!    result := nil;
!end;
Since this method is called a lot of time by our ORM, there is an asm-optimized version of the pascal code above:
!class function TSQLRecord.RecordProps: TSQLRecordProperties;
!asm
!  or eax,eax
!  jz @null
!  mov edx,[eax+vmtAutoTable]
!  or edx,edx
!  jz PropsCreate
!  mov eax,edx
!@null:
!end;
Most of the time, this method will be executed very quickly. In fact, the {\f1\fs20 PropsCreate} global function is called only once, i.e. the first time this {\f1\fs20 RecordProps} method is called.
The {\f1\fs20 TSQLRecordProperties} instance is therefore created within this function:
!function PropsCreate(aTable: TSQLRecordClass): TSQLRecordProperties;
!begin // private sub function makes the code faster in most case
!  if not aTable.InheritsFrom(TSQLRecord) then
!    // invalid call
!    result := nil else begin
!    // create the properties information from RTTI
!    result := TSQLRecordProperties.Create(aTable);
!    // store the TSQLRecordProperties instance into AutoTable unused VMT entry
!    PatchCodePtrUInt(pointer(PtrInt(aTable)+vmtAutoTable),PtrUInt(result));
!    // register to the internal garbage collection (avoid memory leak)
!    GarbageCollector.Add(result);
!  end;
!end;
The {\f1\fs20 GarbageCollector} is a global {\f1\fs20 TObjectList}, which is used to store some global instances, living the whole process time, just like our {\f1\fs20 TSQLRecordProperties} values.
A per-class {\f1\fs20 TSQLRecordProperties} was made therefore available for each kind of {\f1\fs20 TSQLRecord} class.
Even most sophisticated methods of the @*ORM@ (like {\f1\fs20 TSQLRecord. GetJSONValues}) make use of these low-level {\f1\fs20 object} types. In most cases, the {\f1\fs20 GetValue} and {\f1\fs20 SetValue} methods of the {\f1\fs20 TPropInfo object} are used to convert any field value stored inside the current {\f1\fs20 TSQLRecord} instance in or from @*UTF-8@ encoded text.

[SDD-DI-2.1.4]
; SRS-DI-2.1.4 - The framework shall provide some Cross-Cutting components
:Intercepting exceptions
In order to let our {\f1\fs20 TSynLog} logging class, as defined in @!TSynLog,PatchCodePtrUInt!Lib\SynCommons.pas@, intercept all exceptions, we use the low-level global {\f1\fs20 RtlUnwindProc} pointer, defined in {\f1\fs20 System.pas}.
Alas, under {\i Delphi} 5, this global {\f1\fs20 RtlUnwindProc} variable is not existing. The code calls directly the {\f1\fs20 RtlUnWind} Windows API function, with no hope of custom interception.
Two solutions could be envisaged:
- Modify the {\f1\fs20 Sytem.pas} source code, adding the new {\f1\fs20 RtlUnwindProc} variable, just like {\i Delphi} 7;
- Patch the assembler code, directly in the process memory.
The first solution is simple. Even if compiling {\f1\fs20 System.pas} is a bit more difficult than compiling other units, we already made that for our {\i Enhanced RTL units}. But you'll have to change the whole build chain in order to use your custom {\f1\fs20 System.dcu} instead of the default one. And some third-party units (only available in {\f1\fs20 .dcu} form) may not like the fact that the {\i System.pas} interface changed...
So we used the second solution: change the assembler code in the running process memory, to let call our {\f1\fs20 RtlUnwindProc} variable instead of the Windows API.
:One patch to rule them all
The first feature we have to do is to allow on-the-fly change of the assembler code of a process.
In fact, we already use this in order to provide class-level variables, as stated by @SDD-DI-2.1.3@.
We have got the {\f1\fs20 PatchCodePtrUInt} function at hand to change the address of each a {\f1\fs20 RtlUnWind} call.
We'll first define the missing global variable, available since {\i Delphi} 6, for the {\i Delphi} 5 compiler:
!{$ifdef DELPHI5OROLDER}
!// Delphi 5 doesn't define the needed RTLUnwindProc variable :(
!// so we will patch the System.pas RTL in-place
!var
!  RTLUnwindProc: Pointer;
The {\f1\fs20 RtlUnwind} API call we have to hook is defined as such in {\f1\fs20 System.pas}:
!procedure RtlUnwind; external kernel name 'RtlUnwind';
$ 0040115C FF255CC14100     jmp dword ptr [$0041c15c]
The {\f1\fs20 $0041c15c} is a pointer to the address of {\f1\fs20 RtlUnWind} in {\f1\fs20 kernel32.dll}, as retrieved during linking of this library to the main executable process.
The patch will consist in changing this asm call into this one:
$ 0040115C FF25????????     jmp dword ptr [RTLUnwindProc]
Where {\f1\fs20 ????????} is a pointer to the global {\f1\fs20 RTLUnwindProc} variable.
The problem is that we do not have any access to this {\f1\fs20 procedure RtlUnwind} declaration, since it was declared only in the {\f1\fs20 implementation} part of the {\f1\fs20 System.pas} unit. So its address has been lost during the linking process.
So we will have to retrieve it from the code which in fact calls this external API, i.e. from this assembler content:
!procedure       _HandleAnyException;
!asm
$    (...)
$    004038B6 52               push edx  // Save exception object
$    004038B7 51               push ecx  // Save exception address
$    004038B8 8B542428         mov edx,[esp+$28]
$    004038BC 83480402         or dword ptr [eax+$04],$02
$    004038C0 56               push esi  // Save handler entry
$    004038C1 6A00             push $00
$    004038C3 50               push eax
$    004038C4 68CF384000       push $004038cf  // @@returnAddress
$    004038C9 52               push edx
$    004038CA E88DD8FFFF       call RtlUnwind
So we will retrieve the {\f1\fs20 RtlUnwind} address from this very last line.
The {\f1\fs20 E8} byte is in fact the {\i opcode} for the asm {\f1\fs20 call} instruction. Then the called function is stored as an {\i integer} offset, starting from the current pointing value.
The {\f1\fs20 E8 8D D8 FF FF} byte sequence is executed as "{\i call the function available at the current execution address, plus {\f1\fs20 integer($ffffd88d)}}". As you may have guessed, {\f1\fs20 $004038CA+$ffffd88d+5} points to the {\f1\fs20 RtlUnwind} definition.
So here is the main function of this patching:
!procedure Patch(P: PAnsiChar);
!var i: Integer;
!    addr: PAnsiChar;
!begin
!  for i := 0 to 31 do
!    if (PCardinal(P)^=$6850006a) and  // push 0; push eax; push @@returnAddress
!       (PWord(P+8)^=$E852) then begin // push edx; call RtlUnwind
!      inc(P,10); // go to call RtlUnwind address
!      if PInteger(P)^<0 then begin
!        addr := P+4+PInteger(P)^;
!        if PWord(addr)^=$25FF then begin // jmp dword ptr []
!          PatchCodePtrUInt(Pointer(addr+2),cardinal(@RTLUnwindProc));
!          exit;
!        end;
!      end;
!    end else
!    inc(P);
!end;
We will cal this {\f1\fs20 Patch} subroutine from the following code:
!procedure PatchCallRtlUnWind;
!asm
!  mov eax,offset System.@HandleAnyException+200
!  call Patch
!end;
You can note that we need to retrieve the {\f1\fs20 _HandleAnyException} address from asm code. In fact, the compiler does not let access from plain pascal code to the functions of {\f1\fs20 System.pas} having a name beginning with an underscore.
Then the following lines:
!  for i := 0 to 31 do
!    if (PCardinal(P)^=$6850006a) and  // push 0; push eax; push @@returnAddress
!       (PWord(P+8)^=$E852) then begin // push edx; call RtlUnwind
will look for the expected opcode asm pattern in {\f1\fs20 _HandleAnyException} routine.
Then we will compute the position of the {\f1\fs20 jmp dword ptr []} call, via this line:
!        addr := P+4+PInteger(P)^;
After checking that this is indeed a {\f1\fs20 jmp dword ptr []} instruction (expected opcodes are {\f1\fs20 FF 25}), we will simply patch the absolute address with our {\f1\fs20 RTLUnwindProc} procedure variable.
With this code, each call to {\f1\fs20 RtlUnwind} in {\f1\fs20 System.pas} will indeed call the function set by {\f1\fs20 RTLUnwindProc}.
In our case, it will launch the following procedure:
!procedure SynRtlUnwind(TargetFrame, TargetIp: pointer;
!  ExceptionRecord: PExceptionRecord; ReturnValue: Pointer); stdcall;
!asm
!  pushad
!  cmp  byte ptr SynLogExceptionEnabled,0
!  jz   @oldproc
!  mov  eax,TargetFrame
!  mov  edx,ExceptionRecord
!  call LogExcept
!@oldproc:
!  popad
!  pop ebp // hidden push ebp at asm level
!{$ifdef DELPHI5OROLDER}
!  jmp RtlUnwind
!{$else}
!  jmp oldUnWindProc
!{$endif}
!end;
This code will therefore:
- Save the current register context via {\f1\fs20 pushad / popad} opcodes pair;
- Check if {\f1\fs20 TSynLog} should intercept exceptions (i.e. if the global {\f1\fs20 SynLogExceptionEnabled} boolean is true);
- Call our logging function {\f1\fs20 LogExcept};
- Call the default Windows {\f1\fs20 RtlUnwind} API, as expected by the Operating System.

[SDD-DI-2.1.5]
; DI-2.1.5 - The framework shall offer a complete SOA process

You will find out in @!TServiceFactory,TServiceFactoryServer,TServiceFactoryClient,TServiceContainerCLient,TServiceContainerServer,TServiceContainer!Lib\SQlite3\mORMot.pas@ all classes implementing this interface communication.
\graph HierTServiceContainerServer Services implementation classes hierarchy
\TServiceFactoryServer\TServiceFactory
\TServiceFactoryClient\TServiceFactory
\TServiceContainerClient\TServiceContainer
\TServiceContainerServer\TServiceContainer
\
There are two levels of implementation:
- A {\i services catalog}, available in {\f1\fs20 TSQLRest.Services} property, declared as {\f1\fs20 TServiceContainer} (with two {\f1\fs20 inherited} versions, one for each side);
- A {\i service factory} for each interface, declared as {\f1\fs20 TServiceFactory} (also with two {\f1\fs20 inherited} versions, one for each side).
In fact, the {\f1\fs20 TServiceFactory.Create constructor} will retrieve all needed RTTI information of the given interface, i.e. GUID, name and all methods (with their arguments). It will compute the low-level stack memory layout needed at execution to emulate a call of a native {\i Delphi} {\f1\fs20 interface}.  It will use JSON serialization as its internal mean of data persistence and transmission And the corresponding "contract" will be computed from the signature of all interfaces and methods, to validate that both client and server expect the exact same content.
Note that {\f1\fs20 TServiceFactory} will also be used for the stubbing/mocking features of the framework.
On the server side, {\f1\fs20 TServiceFactoryServer.ExecuteMethod} method (and then a nested {\f1\fs20 TServiceMethod.InternalExecute} call) is used to prepare a valid call to the implementation class code from a remote JSON request.
On the client side, a {\f1\fs20 TInterfacedObjectFake} class will be created, and will emulate a regular {\i Delphi} interface call using some on-the-fly asm code generated in the {\f1\fs20 TServiceFactoryClient.Create} constructor.
For technical information about how interfaces are called in {\i Delphi}, see @http://sergworks.wordpress.com/2010/07/06/delphi-interfaces-on-binary-level and the {\f1\fs20 FakeCall} method implementation.
Here is the core of this client-side implementation of the "call stubs":
!  for i := 0 to fMethodsCount-1 do begin
!    fFakeVTable[i+RESERVED_VTABLE_SLOTS] := P;
!    P^ := $68ec8b55; inc(P);                 // push ebp; mov ebp,esp
!    P^ := i; inc(P);                         // push {MethodIndex}
!    P^ := $e2895251; inc(P);                 // push ecx; push edx; mov edx,esp
!    PByte(P)^ := $e8; inc(PByte(P));         // call FakeCall
!    P^ := PtrUInt(@TInterfacedObjectFake.FakeCall)-PtrUInt(P)-4; inc(P);
!    P^ := $c25dec89; inc(P);                 // mov esp,ebp; pop ebp
!    P^ := fMethods[i].ArgsSizeInStack or $900000;  // ret {StackSize}; nop
!    inc(PByte(P),3);
!  end;
Just for fun... I could not resist posting this code here; if you are curious, take a look at the "official" {\f1\fs20 RTTI.pas} or {\f1\fs20 RIO.pas} units as provided by Embarcadero, and you will probably find out that the {\i mORMot} implementation is much easier to follow, and also faster (it does not recreate all the stubs or virtual tables for each wrapper, for instance). :)

[SDD-DI-2.2.1]
; SRS-DI-2.2.1 - The SQLite3 engine must be embedded to the framework
:Implementation
It's worth noting that the {\i Synopse mORMot framework}, whatever its previous name stated ("Synopse SQLite3 framework"), is not bound to {\i SQLite3}.
You can use another database engine for its internal data storage, for example we provide a {\f1\fs20 TSQLRestStorageInMemory} class which implements a fast but limited in-memory database engine.
Therefore, the {\i SQLite3} engine itself is not implemented in the @!TSQLRestServer!Lib\SQLite3\mORMot.pas@ unit, but in dedicated units.
The {\i SQLite3} engine is accessed at two levels:
- A low-level direct access to the {\i SQLite3} library, implemented in @!TSQLite3LibraryDynamic,TSQLite3Library,TSQLRequest.Execute,TSQLDataBase,TSQLTableDB.Create!Lib\SynSQLite3.pas@;
- A low-level statically linked library @!TSQLite3LibraryStatic!Lib\SynSQLite3Static.pas@, embedding the engine within the project executable;
- A possible use of external {\f1\fs20 sqlite3.dll} library, via the {\f1\fs20 TSQLite3LibraryDynamic} class;
- A high-level access, implementing a Client-Side or Server-Side native {\f1\fs20 TSQLRest} descendant using the {\i SQLite3} library for ORM data persistence, in @!TSQLRestServerDB,TSQLRestClientDB!Lib\SQLite3\mORMotSQLite3.pas@.
In addition to those two units, the @!TSQLDBSQLite3Connection,TSQLDBSQLite3Statement,TSQLDBSQLite3ConnectionProperties!Lib\SynDBSQLite3.pas@ unit publishes all features of the {\i SQlite3} database engine to its internal {\f1\fs20 SynDB.pas} fast database access classes, which can be used uncoupled from the rest of the framework (i.e. without ORM).
: Low-Level access to the library
:  SQLite3 API access
Some types are defined in @!TSQLite3Blob,TSQLite3DB,TSQLite3FunctionContext,TSQLite3Statement,TSQLite3Value,TSQLite3ValueArray,TSQLite3Library!Lib\SynSQLite3.pas@ to map the types used by {\i SQLite3}: {\f1\fs20 TSQLite3DB, TSQLite3Statement, TSQLite3Blob, TSQLite3Value, TSQLite3FunctionContext}, which are mapped to a {\f1\fs20 PtrUInt}, i.e. an unsigned integer matching the current pointer size. This is the {\i handle} type exposed by the {\i SQLite} API.
Then most C-language interface to {\i SQLite} has been converted into pure {\i Delphi} external {\f1\fs20 function} or {\f1\fs20 procedure} calls (see @http://www.sqlite.org/c3ref/intro.html for a complete reference). The conversion rule was to match the API name (all {\f1\fs20 sqlite3: TSQLite3Library} identifiers), then provide the most {\i Delphi}-standard access to the parameters: for instance, we use standard Integer/Int64/PUTF8Char types, or a {\f1\fs20 var} declaration instead of a C pointer.
A global variable is used to access the {\i SQLite3} engine shared instance:
!var
!  /// global access to linked SQLite3 library API calls
!  // - you should call sqlite3.open() instead of sqlite3_open() for instance
!  // - points either to the statically linked sqlite3.obj, or to an external
!  // library (e.g. sqlite3.dll under Windows)
!  // - your project should use EITHER SynSQLite3Static unit OR create a
!  // TSQLite3LibraryDynamic instance:
!  // ! FreeAndNil(sqlite3); // release any previous instance
!  // ! sqlite3 := TSQLite3LibraryDynamic.Create;
!  sqlite3: TSQLite3Library;
All the framework code will use this {\f1\fs20 sqlite3} global instance - using e.g. {\f1\fs20 qlite3.open()} instead of {\f1\fs20 sqlite3_open()} - but some custom low-level code may use several {\i SQLite3} implementations at once.
: High level access
Some {\i Delphi} classes are introduced to manage all calls and statements to C-language interface to {\i SQLite}, mapping all {\f1\fs20 sqlite3*} functions and methods to object-oriented methods.
The @!TSQLTableDB,TSQLRequest,TSQLDataBase,TSQLBlobStream,ESQLException!Lib\SynSQLite3.pas@ unit defines the following classes:
- {\f1\fs20 ESQLException} is a custom {\i SQLite3} dedicated Exception type;
- {\f1\fs20 TSQLDataBase} is a simple wrapper for direct {\i SQLite3} database manipulation;
- {\f1\fs20 TSQLRequest} encapsulates a {\i SQLite3} request;
- {\f1\fs20 TSQLTableDB} executes a @*SQL@ statement in the local {\f1\fs20 SQLite3} database engine, and get result in memory, as JSON content;
- {\f1\fs20 TSQLBlobStream} is available to access to a {\i SQLite3} BLOB Stream.
Those database access types are then used by the following Client-Server @*REST@ful classes, to implement {\i SQLite3} storage for persistence of our @*ORM@ (the so called objects hibernation) in @!TSQLRestClientDB,TSQLRestServerDB!Lib\SQLite3\mORMotSQLite3.pas@:
- {\f1\fs20 TSQLRestClientDB} implements a REST client with direct access to a {\i SQLite3} database, that is without the Client-Server aspect of the framework;
- {\f1\fs20 TSQLRestServerDB} can be used to implement a REST server using {\i SQLite3} as its storage engine.
In most projects, you should not have to use those {\f1\fs20 TSQLDatabase / TSQLRequest} objects, but rather rely either on ORM classes (from {\f1\fs20 mORMot.pas}) or the more generic {\f1\fs20 SynDB.pas} classes (from {\f1\fs20 SynDBSQlite3.pas}).

[SDD-DI-2.2.2]
; SRS-DI-2.2.2 - The framework libraries, including all its {\i SQLite3} related features, must be tested using Unitary testing
:Implementation
Some @*test@s classes have been developed, which methods cover most aspect of the framework:
\graph HierTSynTestCase TSynTestCase classes hierarchy
\TTestSynopsePDF\TSynTestCase
\TTestMemoryBased\TTestSQLite3Engine
\TTestFileBasedWAL\TTestFileBased
\TTestFileBased\TTestSQLite3Engine
\TTestSQLite3Engine\TSynTestCase
\TTestLowLevelTypes\TSynTestCase
\TTestLowLevelCommon\TSynTestCase
\TTestCryptographicRoutines\TSynTestCase
\TTestCompression\TSynTestCase
\TTestClientServerAccess\TSynTestCase
\TTestServiceOrientedArchitecture\TSynTestCase
\TTestBigTable\TSynTestCase
\TTestBasicClasses\TSynTestCase
\TSynTestCase\TSynTest
rankdir=LR;
\
Those classes are implemented in @!TTestLowLevelCommon,TTestLowLevelTypes,TTestBasicClasses,TTestSQLite3Engine,TTestFileBased,TTestMemoryBased,TTestFileBasedWAL,TTestClientServerAccess!Lib\SynSelfTests.pas@ units.

[SDD-DI-2.2.3]
; SRS-DI-2.2.3 - The framework shall be able to access any external database, via OleDB or direct access for Oracle (OCI) or SQLite3 (for external database files)
:SynDB classes
The @SAD@ document detailed the architecture, and main implementation part of the database-agnostic features of the framework.
:Faster late-binding
For several units of our framework, we allow {\i late-binding} of data values, using a {\f1\fs20 variant} and direct named access to properties:
- In {\i SynCommons}, we defined our {\f1\fs20 TDocVariant} custom variant type, able to store any JSON/BSON document-based content;
- In {\i SynBigTable}, we use the {\f1\fs20 TSynTableVariantType} custom variant type, as defined in {\i SynCommons};
- In {\i SynDB}, we defined a {\f1\fs20 TSQLDBRowVariantType}, ready to access any column of a RDBMS data result set row;
- In {\i mORMot}, we allow access to {\f1\fs20 TSQLTableRowVariantType} column values.
It's a very convenient way of accessing result rows values. Code is still very readable, and safe at the same time.
For instance, we can write:
!var V: variant;
! ...
!  TDocVariant.New(V); // or slightly slower V := TDocVariant.New;
!  V.name := 'John';
!  V.year := 1972;
!  // now V contains {"name":"john","year":1982}
This is just another implementation of KISS design in our framework.
: Speed issue
But, in practice, this approach is slower, due to the current implementation of the {\i Delphi} RTL.
Late-binding uses the internal mechanism used for {\i Ole Automation}, here to access column content as if column names where native object properties. There is plenty of space for speed improvement here.
So, how does the variant type used by {\i Ole Automation} and our custom variant types (i.e. {\f1\fs20 TSynTableVariantType} or {\f1\fs20 TSQLDBRowVariantType}) handle their properties access?
Behind the scene, the {\i Delphi} compiler calls the {\f1\fs20 DispInvoke} function, as defined in the {\i Variant.pas} unit.
The default implementation of this {\f1\fs20 DispInvoke} is some kind of slow:
- It uses a {\f1\fs20 TMultiReadExclusiveWriteSynchronizer} under {\i Delphi} 6, which is a bit over-sized for its purpose: since {\i Delphi} 7, it uses a lighter critical section;
- It makes use of {\f1\fs20 @*WideString@} for string handling (not at all the better for speed), and tends to define a lot of temporary string variables;
- For the getter method, it always makes a temporary local copy during process, which is not useful for our classes.
: Fast and furious
So we rewrite the {\f1\fs20 DispInvoke} function with some enhancements in mind:
- Will behave exactly the same for other kind of variants, in order to avoid any compatibility regression, especially with {\i Ole Automation};
- Will quick intercept our custom variant types (as registered via the global {\f1\fs20 SynRegisterCustomVariantType} function), and handle those with less overhead: no critical section nor temporary {\f1\fs20 WideString} allocations are used.
: Implementation
Here is the resulting code, from our {\i SynCommons} unit:
!procedure SynVarDispProc(Result: PVarData; const Instance: TVarData;
!      CallDesc: PCallDesc; Params: Pointer); cdecl;
!const DO_PROP = 1; GET_PROP = 2; SET_PROP = 4;
!var i: integer;
!    Value: TVarData;
!    Handler: TCustomVariantType;
!begin
!  if Instance.VType=varByRef or varVariant then // handle By Ref variants
!    SynVarDispProc(Result,PVarData(Instance.VPointer)^,CallDesc,Params) else begin
!    if Result<>nil then
!      VarClear(Variant(Result^));
!    case Instance.VType of
!    varDispatch, varDispatch or varByRef,
!    varUnknown, varUnknown or varByRef, varAny:
!       // process Ole Automation variants
!        if Assigned(VarDispProc) then
!          VarDispProc(pointer(Result),Variant(Instance),CallDesc,@Params);
!    else begin
!      // first we check for our own TSynInvokeableVariantType types
!      if SynVariantTypes<>nil then
!      for i := 0 to SynVariantTypes.Count-1 do
!        with TSynInvokeableVariantType(SynVariantTypes.List[i]) do
!        if VarType=TVarData(Instance).VType then
!        case CallDesc^.CallType of
!        GET_PROP, DO_PROP: if (Result<>nil) and (CallDesc^.ArgCount=0) then begin
!          IntGet(Result^,Instance,@CallDesc^.ArgTypes[0]);
!          exit;
!        end;
!        SET_PROP: if (Result=nil) and (CallDesc^.ArgCount=1) then begin
!          ParseParamPointer(@Params,CallDesc^.ArgTypes[0],Value);
!          IntSet(Instance,Value,@CallDesc^.ArgTypes[1]);
!          exit;
!        end;
!        end;
!      // here we call the default code handling custom types
!      if FindCustomVariantType(Instance.VType,Handler) then
!        TSynTableVariantType(Handler).DispInvoke(
!          {$ifdef DELPHI6OROLDER}Result^{$else}Result{$endif},
!          Instance,CallDesc,@Params)
!      else raise EInvalidOp.Create('Invalid variant invoke');
!    end;
!    end;
!  end;
!end;
Our custom variant types have two new virtual protected methods, named {\f1\fs20 IntGet/IntSet}, which are the getter and setter of the properties. They will to the property process, e.g. for our {\i OleDB} column retrieval:
!procedure TSQLDBRowVariantType.IntGet(var Dest: TVarData;
!  const V: TVarData; Name: PAnsiChar);
!var Rows: TSQLDBStatement;
!begin
!  Rows := TSQLDBStatement(TVarData(V).VPointer);
!  if Rows=nil then
!    EOleDBException.Create('Invalid SQLDBRowVariant call');
!  Rows.ColumnToVariant(Rows.ColumnIndex(RawByteString(Name)),Variant(Dest));
!end;
As you can see, the returned variant content is computed with the following method:
!function TOleDBStatement.ColumnToVariant(Col: integer;
!  var Value: Variant): TSQLDBFieldType;
!const FIELDTYPE2VARTYPE: array[TSQLDBFieldType] of Word = (
!  varEmpty, varNull, varInt64, varDouble, varCurrency, varDate,
!  {$ifdef UNICODE}varUString{$else}varOleStr{$endif}, varString);
!var C: PSQLDBColumnProperty;
!    V: PColumnValue;
!    P: pointer;
!    Val: TVarData absolute Value;
!begin
!  V := GetCol(Col,C);
!  if V=nil then
!    result := ftNull else
!    result := C^.ColumnType;
!  VarClear(Value);
!  Val.VType := FIELDTYPE2VARTYPE[result];
!  case result of
!    ftInt64, ftDouble, ftCurrency, ftDate:
!      Val.VInt64 := V^.Int64; // copy 64-bit content
!    ftUTF8: begin
!      Val.VPointer := nil;
!      if C^.ColumnValueInlined then
!        P := @V^.VData else
!        P := V^.VAnsiChar;
!      SetString(SynUnicode(Val.VPointer),PWideChar(P),V^.Length shr 1);
!    end;
!    ftBlob: begin
!      Val.VPointer := nil;
!      if C^.ColumnValueInlined then
!        P := @V^.VData else
!        P := V^.VAnsiChar;
!      SetString(RawByteString(Val.VPointer),PAnsiChar(P),V^.Length);
!    end;
!    end;
!end;
This above method will create the variant content without any temporary variant or string. It will return TEXT ({\f1\fs20 ftUTF8}) column as {\f1\fs20 @*SynUnicode@}, i.e. into a generic {\f1\fs20 @*WideString@} variant for pre-Unicode version of {\i Delphi}, and a generic {\f1\fs20 UnicodeString} (={\f1\fs20 string}) since {\i Delphi} 2009. By using the fastest available native Unicode {\f1\fs20 string} type, you will never loose any Unicode data during char-set conversion.
: Hacking the VCL
In order to enable this speed-up, we'll need to change each call to {\f1\fs20 DispInvoke} into a call to our custom {\f1\fs20 SynVarDispProc} function.
With {\i Delphi} 6, we can do that by using {\f1\fs20 GetVariantManager   /SetVariantManager} functions, and the following code:
!    GetVariantManager(VarMgr);
!    VarMgr.DispInvoke := @SynVarDispProc;
!    SetVariantManager(VarMgr);
But since {\i Delphi} 7, the {\f1\fs20 DispInvoke} function is hard-coded by the compiler into the generated asm code. If the {\i Variants} unit is used in the project, any late-binding variant process will directly call the {\f1\fs20 _DispInvoke} private function of {\f1\fs20 Variants.pas}.
First of all, we'll have to retrieve the address of this {\f1\fs20 _DispInvoke}. We just can't use {\f1\fs20 _DispInvoke} or {\f1\fs20 DispInvoke} symbol, which is not exported by the {\i Delphi} linker... But this symbol is available from asm!
So we will first define a pseudo-function which is never called, but will be compiled to provide a pointer to this {\f1\fs20 _DispInvoke} function:
!procedure VariantsDispInvoke;
!asm
!  call Variants.@DispInvoke;
!end;
Then we'll compute the corresponding address via this low-level function, the asm {\f1\fs20 call} opcode being {\f1\fs20 $E8}, followed by the relative address of the sub-routine:
!function GetAddressFromCall(AStub: Pointer): Pointer;
!begin
!  if AStub=nil then
!    result := AStub else
!  if PBYTE(AStub)^ = $E8 then begin
!    Inc(PtrInt(AStub));
!!    Result := Pointer(PtrInt(AStub)+SizeOf(integer)+PInteger(AStub)^);
!  end else
!    Result := nil;
!end;
And we'll patch this address to redirect to our own function:
! RedirectCode(GetAddressFromCall(@VariantsDispInvoke),@SynVarDispProc);
The resulting low-level asm will just look like this at the call level:
$!TestOleDB.dpr.28: assert(Copy(Customer.AccountNumber,1,8)='AW000001');
$00431124 8D45D8           lea eax,[ebp-$28]
$00431127 50               push eax
$00431128 6828124300       push $00431228
$0043112D 8D45E8           lea eax,[ebp-$18]
$00431130 50               push eax
$00431131 8D45C4           lea eax,[ebp-$3c]
$00431134 50               push eax
$!00431135 E86ED1FDFF       call @DispInvoke
It will therefore call the following hacked function:
$0040E2A8 E9B3410100       jmp SynVarDispProc
$0040E2AD E853568B5D       call +$5d8b5653
$... (previous function content, never executed)
That is, it will jump ({\f1\fs20 jmp}) to our very own {\f1\fs20 SynVarDispProc}, just as expected.
In fact, the resulting code is very close to a direct {\f1\fs20 ISQLDBRows.Column['AccountNumber']} call. Using {\i late-binding} can be both fast on the execution side, and easier on the code side.
:   Delphi XE2 and up
Since {\i Delphi} XE2, some modifications were introduced to the official {\f1\fs20 DispInvoke()} RTL implementation:
- A new {\f1\fs20 varUStrArg} kind of parameter has been defined, which will allow to transmit {\f1\fs20 UnicodeString} property values;
- All text property values will be transmitted as BSTR {\f1\fs20 / WideString / varOleStr} variants to the invoked variant type;
- All textual property names were normalized to be in UPPERCASE.
The first modification does make sense, and was indeed a welcome fix for an Unicode version of {\i Delphi}. It should have been as such since {\i Delphi} 2009.
Temporary conversion to {\f1\fs20 WideString} does make sense in the COM / OLE world, but is an awfull performance bottleneck in the pure {\i Delphi} realm, i.e. when using late-binding with custom type of variants (as for all our custom variant types). This may be a noticeable speed penalty, in comparison to previous versions of the compiler.
Last but not least, the conversion to uppercase is a bug. For instance, the following code won't work as expected since {\i Delphi} XE2:
!var V: variant;
! ...
!  TDocVariant.New(V); // or slightly slower V := TDocVariant.New;
!  V.name := 'John';
!  V.year := 1972;
!  // before Delphi XE2, V contains {"name":"john","year":1982} - as expected
!  // since Delphi XE2,  V contains {"NAME":"john","YEAR":1982} - sounds like a bug, doesn't it?
This sounds indeed like an awfull regression.
Since revision 1.18 of the framework, the patch described in this {\i Design Input} has been modified for {\i Delphi} XE2 and up, as such:
- It will handle {\f1\fs20 varUStrArg} kind of parameter as exepcted;
- It will avoid any temporary conversion to {\f1\fs20 WideString} for textual values;
- It will by-pass the property name change into uppercase.
As soon as you define {\i SynCommons} in any of your program's uses class, our hooked {\f1\fs20 DispInvoke()} will take place, and identify any of our {\f1\fs20 TSynInvokeableVariantType} classes. It will by-pass the performance bottleneck of the default RTL implementation, and also fix the uppercase conversion of the property name.
Of course, if this variant is not a {\f1\fs20 TSynInvokeableVariantType} instance (e.g. any {\i Ole Automation} call), the regular {\f1\fs20 TInvokeableVariantType.DispInvoke()} method as defined in {\f1\fs20 Variants.pas} will be executed, to maintain the best compatibility possible.

[SDD-DI-2.3]
:SynFile main Demo
The @SAD-SynFile@ section of the associated @SAD@ has already detailed the architecture and the code used to produce a full featured application, including the User Interface generation.
Please refer to these pages for sample code and general explanation about this feature of the framework.

[SDD-DI-2.3.1.1]
; SRS-DI-2.3.1.1 - Database Grid Display, providing data in the Client Application
:Implementation
A standard {\f1\fs20 TDrawGrid} can be associated to a {\f1\fs20 TSQLTable} instance by using a {\f1\fs20 TSQLTableToGrid} object, as defined in the @!TSQLTableToGrid.Create!Lib\SQLite3\mORMotUI.pas@:
- Just call {\f1\fs20 TSQLTableToGrid.Create(Grid,Table)} to initiate the association;
- The Table will be released when no longer necessary;
- Any former association by {\f1\fs20 TSQLTableToGrid.Create()} will be overridden;
- Handle Unicode, auto column size, field sort, incremental key lookup, optional hide ID;
- {\i Ctrl + click} on a cell to display its full Unicode content.
For instance, here is how the @!TSQLLister.Create!Lib\SQLite3\mORMotToolBar.pas@ unit creates a grid for every {\f1\fs20 TSQLRecord} class it refers to:
!constructor TSQLLister.Create(aOwner: TComponent; aClient: TSQLRestClientURI;
!  (...)
!!  fTableToGrid := TSQLTableToGrid.From(fGrid);
!  if fTableToGrid=nil then begin
!    // this Grid has no associated TSQLTableToGrid -> create default one
!    if fClient.InheritsFrom(TSQLRestClientURI) then
!      C := TSQLRestClientURI(fClient) else
!      C := nil;
!!    fTableToGrid := TSQLTableToGrid.Create(fGrid,aTable,C);
!    if aIDColumnHide then
!!      fTableToGrid.IDColumnHide;
!  end;
!  fTableToGrid.OnRightClickCell := OnRightClickCell;
!  TableToGrid.OnValueText := aOnValueText;
!  fGrid.DefaultDrawing := false; // we force full redraw
!  TableToGrid.OnDrawCellBackground := OnDrawCellBackground;
!  TableToGrid.OnSelectCell := OnSelectCell;
!  (...)
All the process will be done in an automated manner, using the methods of the {\f1\fs20 TDrawGrid} component.
The current implementation is very fast, since the data is taken directly from the {\f1\fs20 TSQLTable} content. A grid with more than 200,000 rows is displayed with no delay. All content is converted into pure text, according to the @*RTTI@ information associated with the {\f1\fs20 TSQLTable} columns. If it was created as a {\f1\fs20 TSQLTableJSON}, from an @*ORM@ call of the framework, it will contain the RTTI information for each column. For instance, time and date will be displayed with the current internationalization settings, from either @*ISO 8601@ encoded text (for {\f1\fs20 @*TDateTime@} or @*TDateTimeMS@ published property) or our optimized {\f1\fs20 Int64} format (for {\f1\fs20 @*TTimeLog@ / @*TModTime@ / @*TCreateTime@} published property).

[SDD-DI-2.3.1.2]
; SRS-DI-2.3.1.2 - Toolbar creation from code, using RTTI
:Implementation
: Rendering
The current implementation of the framework User Interface generation handles two kind of rendering:
- Native VCL components;
- Proprietary TMS components.
You can select which set of components are used, by defining - globally to your project (i.e. in the {\i Project/Options/Conditionals} menu) - the {\f1\fs20 USETMSPACK} conditional. If it is not set (which is by default), it will use VCL components.
: Ribbon-like toolbars
As stated by the @SAD-SynFile@ section of the associated @SAD@, ribbon-like toolbars can be generated by using the {\f1\fs20 TSQLRibbon} class, as defined in @!TSQLRibbon.Create,TSQLRibbonTab,TSQLLister,TSQLCustomToolBar.Init!Lib\SQLite3\mORMotToolBar.pas@.
This class will use one {\f1\fs20 TSQLRibbonTab} instance per {\f1\fs20 TSQLRecord} class type it handles, displayed on its own ribbon page, with an associated {\f1\fs20 TDrawGrid} instance and a {\f1\fs20 TGDIPages} report, via a corresponding {\f1\fs20 TSQLLister} instance. Parameters provided from code to the {\f1\fs20 TSQLRibbon. Create} method can customize the toolbar content on purpose. Actions will be provided as an enumeration type, and button captions will be extracted by {\i Un @*Camel@ Casing} of each @*enumerate@d value, using @*RTTI@.
: Stand-alone toolbars
A {\f1\fs20 TSQLCustomToolBar} object can be used to create some generic toolbars, with just some icons and actions on screen, with no reference to any associated {\f1\fs20 TSQLRecord} class. See for instance this sample code:
!procedure TMainLogView.FormCreate(Sender: TObject);
!begin
!  FToolBar.Init(self,TypeInfo(TLogViewAction),ActionClick,ImageList,'');
!  FToolBar.AddToolBar('Test')
!end;
The above lines will create a panel on the owner form, with a toolbar containing one button per each {\f1\fs20 TLogViewAction} element. Icons will be taken from the supplied {\f1\fs20 ImageList} component, and the {\f1\fs20 ActionClick} event handler will be called when a button is pressed.

[SDD-DI-2.3.1.3]
; SRS-DI-2.3.1.3 - Internationalization (i18n) of the whole User Interface
:Implementation
The @!TLanguage,TLanguageFile.Create,S2U,U2S,TLanguageFile.StringToUTF8,TLanguageFile.TimeToText,TLanguageFile.DateToText,TLanguageFile.DateTimeToText,TLanguageFile.UTF8ToString,TLanguageFile.Translate,_!Lib\SQLite3\mORMoti18n.pas@ unit is able to handle both Internationalization (i18n) and Localization (L10n).
The {\f1\fs20 TLanguageFile} class is able to retrieve a custom list of text, and use it for all {\f1\fs20 resourcestring} and screen captions. The global {\f1\fs20 _()} function, or the {\f1\fs20 Translate} method of the {\f1\fs20 TLanguageFile} class can be used to translate any English text into the corresponding language.
The generic {\f1\fs20 string} type is used when some text is to be displayed on screen. Dedicated {\f1\fs20 U2S} and {\f1\fs20 S2U} functions, or even better the {\f1\fs20 UTF8ToString} and {\f1\fs20 StringToUTF8} methods of a {\f1\fs20 TLanguageFile} instance can be used for proper conversion.
Localization is performed via some dedicated methods of the {\f1\fs20 TLanguageFile} class, like {\f1\fs20 DateToText, DateTimeToText, TimeToText}.

[SDD-DI-2.3.2]
; SRS-DI-2.3.2 - A reporting feature, with full preview and export as PDF or TXT files, must be integrated
:Implementation
The @!TGDIPages!Lib\SQLite3\mORMotReport.pas@ unit implements a reporting component named {\f1\fs20 TGDIPages}, with full preview and {\f1\fs20 txt/pdf} export.
Anti-aliased drawing is using the @!TGDIPlus.DrawAntiAliased!Lib\SynGdiPlus.pas@ unit, and the {\f1\fs20 TGDIPlus. DrawAntiAliased} method.
The pdf export itself is implemented via the @!TPdfDocument,TPdfCanvas.RenderMetaFile!Lib\SynPdf.pas@ unit, via a {\f1\fs20 TPdfDocument} component: every page content (in fact, a {\f1\fs20 TMetaFile} instance) is rendered via the {\f1\fs20 TPdfCanvas. RenderMetaFile} method.

[VV]
Owner=SRS
Order=SRS
DisplayName=V&V Plan
DocName=V&V Plan
Name=Software Validation and Verification Plan
Purpose=Define the testing required for the updates to the mORMot Framework software along with the testing responsibilities
; just the message for 'Goal:' in the Software Verification Plan section
PreparedBy=Arnaud Bouchez
ReviewedBy=
ApprovedBy=
Revision=1.8
RevisionDate=
RevisionDescription=Initial Version
; Revision* multiple revision Table: ignored values are taken from current, older below
DocumentFrontPage=ProjectDetails,Warning,PeopleDetails,RevisionDetails,AddPurpose
WriteTableOfContent=Yes
WriteSummaryOf=Test
; WriteSummaryOf=Test -> VV document will have 2 parts: [VV] body as introduction, then a list of all documents divided by [Test-*].Description=.., with [Test].Goal,[Test].DocName, and associated [SRS-*] (Owner=SRS) sections

:Introduction
: Purpose
This @VV@ applies to the upgrade software for the mORMot Framework, implementing the Software part of the @DI@.
Its activities cover the modifications to the software as described in the @SRS@.
: Scope
The software supplied with the instrument is divided in the following parts, as specified by the @SAD@:
...
: Risks and Contingencies
Modifications to the software are required to resolve defects or add additional approved functionality. For each new version of software the risks must be assessed if the complete V&V plan is not followed. At a minimum the following must be documented: description of change, potential impact on software, minimal testing of software that must be performed and a summary of the testing. For additional risks see the @Risk@.
: Approach
The overall approach to testing will be functionality (black box) testing. Some test will be made with specific tools (software or debugger). The comprehensiveness of the testing will be evaluated by the completion of the test summary where the features are listed.
Any software defects will be reported to the developer and added to the bug tracking database. Defects will be reviewed and resolved based on the severity, occurrence and customer impact assigned to each defect.
: Item Pass/Fail criteria
{\i Pass/Fail} criteria will be determined based on the software requirements definition.
: Test results
Test results will be documented on the test protocol summary sheet. Any defects reported during testing will also be referenced on the test protocol summary.
: Test reports
Test Reports will be compiled from the Test Plan, Procedures and Test Results.  The purposes of the Test Reports are to summarize the test protocols and results and draw a conclusion regarding the validation of the {\i Synopse mORMot Framework} to meet its design goals.
The Test Reports will contain the following information:
- Software Version Tested;
- Summary of test results;
- Summary of observations not included in this V&V plan;
- Recommendations;
- List of the features to be tested showing the test procedure used to test each specification, version tested and the pass/fail determination;
- List of each test procedure showing the date of testing, tester, pass/fail determination and reference to any defect observed in testing;
- Software Problem Reports. List all defects reported during testing with the priority and open/closed status.
: Software Verification Plan
The Test Reports are divided into several document files, as listed in the {\i Software Verification Plan} below.
The {\i Software Verification Plan} layout follows the main sections of the @DI@:
\LayoutPage
: Responsibilities
This document is intended to be reviewed by QA team.
It is the responsibility of the Software V&V person or team to:
- Follow the software V&V protocols prepared in support of this V&V plan and to document the results;
- Complete all parts of a protocol in which a selection, such as Pass or Fail is requested or explain why a determination could not be made;
- Generate a V&V report noting compliance and deviance of the measured or observed from the expected and to submit it for review.
It is the responsibility of all Software Managers, Project Manager, or their designee to review and approve the software V&V plan and the software report.
\page
:Software Verification Plan

[Test]
Owner=SRS
Order=SRS
; Owner: [Test-*] -> * reference; Order=Test -> [Test-*] have sub items
Name=Test protocols
ItemName=Test protocol
DocName=Test
Purpose=Describe all Test protocols with specific pass/fail criteria
PreparedBy=Arnaud Bouchez
ReviewedBy=
ApprovedBy=
Revision=1.8
RevisionDate=
RevisionDescription=Initial Version
; Revision* multiple revision Table: ignored values are taken from current, older below
; [Test-*] sections contain details for each SRS, [Test-SER-03] or [Test-DI-4.7.6] e.g.
; [Test-*] are displayed as they appear in the [SRS-*] sections
; all individual Test documents can be created with the [Tests] section
; a global 'Test protocols.doc' can also be created, containing all Test reports in one big file, with corresponding options
;WriteRisk=Yes
; global 'Test protocols.doc' will contain corresponding Risk assessment
BodyIsTest=Yes
; so any [Test-*] will have a special format: | at the beginning of the line, like |Actions[|Expected Results[|Observations]] - same as a table, but with no |% before and after, and possibly missing |
; -> text between | has to use \line to between paragraph/lines
; -> a line with only | is a separator between tests: a new table will be printed
; -> if any value is entered in 'Expected Results', a Pass/Fail message will be added in Validation
DocumentFrontPage=ProjectDetails,Warning,PeopleDetails,RevisionDetails,AddPurpose

:Introduction
This @Test@ regroups all the {\i Test protocols} in an unique document. It may be convenient to have all the test procedures in the same file, for review purpose, e.g.
Every @SRS@ item is listed with an abstract of its implementation, then its specific protocol is written, following the main sections the @DI@:
\LayoutPage
;The numerical Risk evaluation, as stated in @Risk@, is written again for every Design Input item, according to the {\i Risk Assessment Scale} table shown on page 2.

[Tests]
Owner=SRS
Order=SRS
Name=Test report
Purpose=Create all Tests protocols documents
; these Purpose= value will be used in the menu item hint
DocByDescription=Test
; -> individual documents divided by [Test-*].Description=.. can be created, with layout in [Tests]
; -> a last page is added, named 'Summary Sheet', taking executable versions and name in [SAD-*].Source= (SAD is TProject.ParseDoc)
SubDocFrontPage=TestDetails,RevisionDetails,AddPurpose
; additionnal front page values are taken from [Test-*].Requirements=.. and [Test-*].Notes=..
PreparedBy=Arnaud Bouchez
; // may be overwritten in [Test-*]

; body of this section is the last page summary sheet
{\ul{\b START TIME:}}    {\f1 __________________    }{\ul{\b END TIME:}}    {\f1 __________________}\line
{\ul{\b RELEVANT TESTING INFORMATION}}\line{\f1 _______________________________________________________________\line _______________________________________________________________\line _______________________________________________________________}\line
{\ul{\b DEVIATIONS FROM PROCEDURE}}\line {\f1 _______________________________________________________________\line _______________________________________________________________\line _______________________________________________________________}\line
{\ul{\b ATTACHMENTS LISTING}}
\par
\par
\par
{\b [  ]  MEETS SPECIFICATIONS}
{\b [  ]  DOES NOT MEET SPECIFICATIONS}\line
{\b PROCEDURE PERFORMED BY:\line{\f1  _____________________________________}}
{\b DATE:}{\f1  _____________}\line
{\b REVIEW AND APPROVAL BY:\line{\f1  _____________________________________}}
{\b DATE:}{\f1  _____________}

[SoftwareVersion]

:Synopse mORMot Framework
|%23%18%15%44
|\b File Name|Date|Version|Description\b0
|...|...|...|...
|%

[KnownIssues]

|%15%85
|\b Request|Description\b0
|...|...
|%

[SoftwareHistory]

:mORMot Framework
|%8%17%75
|\b Version|Date|Remarks\b0
|1.00|...|...
|%

[GPL]
:GNU General Public License
\include gpl-3.0.txt

[License]
:34License
: Three Licenses Model
The framework source code is licensed under a disjunctive three-@**license@ giving the user the choice of one of the three following sets of free software/open source licensing terms:
- {\i @*Mozilla Public License@}, version 1.1 or later (MPL);
- {\i GNU @*General Public License@}, version 2.0 or later (GPL);
- {\i GNU @*Lesser General Public License@}, version 2.1 or later (LGPL), with {\i linking exception} of the {\i FPC modified LGPL}.
{\i FPC modified LGPL} is the {\i Library GNU General Public License} with the following modification:\line As a special exception of the LGPL, the copyright holders of this library give you permission to link this library with independent modules to produce an executable, regardless of the license terms of these independent modules, and to copy and distribute the resulting executable under terms of your choice, provided that you also meet, for each linked independent module, the terms and conditions of the license of that module. An independent module is a module which is not derived from or based on this library. If you modify this library, you may extend this exception to your version of the library, but you are not obligated to do so. If you do not wish to do so, delete this exception statement from your version.
This allows the use of the framework code in a wide variety of software projects, while still maintaining intellectual rights on library code.
In short:
- For GPL projects, use the GPL license - see @http://www.gnu.org/licenses/gpl-2.0.html
- For LGPL projects, use the LGPL license - see @http://www.gnu.org/licenses/lgpl-2.1.html
- For commercial projects, use the MPL License - see @http://www.mozilla.org/MPL/MPL-1.1.html - which is the most permissive, or the FPC modified LGPL license, thanks to its linking exception - see @http://wiki.freepascal.org/modified_LGPL
: Publish modifications and credit for the library
In all cases, any modification made to this source code {\b should} be published by any mean (e.g. a download link), even in case of MPL. If you need any additional feature, use the forums and we may introduce a patch to the main framework trunk.
You do not have to pay any fee for using our MPL/GPL/LGPL libraries.
But please do not forget to put somewhere in your credit window or documentation, a link to @https://synopse.info if you use any of the units published under this tri-license.
For instance, if you select the MPL license, here are the requirements:
- You accept the license terms with no restriction - see @http://www.mozilla.org/MPL/2.0/FAQ.html for additional information;
- You have to publish any modified unit (e.g. {\f1\fs20 SynTaskDialog.pas}) in a public web site (e.g. {\f1\fs20 http://SoftwareCompany.com/MPL}), with a description of applied modifications, and no removal of the original license header in source code;
- You make appear some notice available in the program (About box, documentation, online help), stating e.g.\line {\i This software uses some third-party code of the Synopse mORMot framework (C) 2022 Arnaud Bouchez - {\f1\fs20 https://synopse.info} - under Mozilla Public License 1.1; modified source code is available at {\f1\fs20 http://SoftwareCompany.com/MPL}.}
: Derivate Open Source works
If you want to include part of the framework source code in your own open-source project, you may publish it with a comment similar to this one (as included in the great {\i DelphiWebScript} project by Eric Grange - @http://code.google.com/p/dwscript ):
${
$    Will serve static content and DWS dynamic content via http.sys
$    kernel mode high-performance HTTP server (available since XP SP2).
$    See http://blog.synopse.info/post/2011/03/11/HTTP-server-using-fast-http.sys-kernel-mode-server
$    WARNING: you need to first register the server URI and port to the http.sys stack.
$    That is, run the application at least once as administrator.
$
$    Sample based on official mORMot's sample
$    "SQLite3\Samples\09 - HttpApi web server\HttpApiServer.dpr"
$
$    Synopse mORMot framework. Copyright (C) 2022 Arnaud Bouchez
$      Synopse Informatique - https://synopse.info
$
$    Original tri-license: MPL 1.1/GPL 2.0/LGPL 2.1
$
$    You will need at least the following files from mORMot framework
$    to be available in your project path:
$    - SynCommons.pas
$    - Synopse.inc
$    - SynLZ.pas
$    - SynZip.pas
$    - SynCrtSock.pas
$    - SynWinWock.pas
$    https://synopse.info/fossil/wiki?name=Downloads
$
$}
Note that this documentation is under GPL license only, as stated in this document front page.
: Commercial licenses
Even though our libraries are Open Source with permissive licenses, some users want to obtain a license anyway. For instance, you may want to hold a tangible legal document as evidence that you have the legal right to use and distribute your software containing our library code, or, more likely, your legal department tells you that you have to purchase a license.
If you feel like you really have to purchase a license for our libraries, {\i Synopse}, the company that employs the architect and principal developer of the libray, will sell you one. Please contact us directly for a contract proposal.

[SCRS]
Owner=SRS
;Order=SRS
; no Order= specified, so that the body of this [SCRS] document section will be written as plain text only
Name=Software Change Request Summary Form
Purpose=Cross-reference all software changes
ItemName=SCRS
PreparedBy=Arnaud Bouchez
ReviewedBy=
ApprovedBy=
Revision=1.8
RevisionDate=
RevisionDescription=Initial Version
; Revision* multiple revision Table: ignored values are taken from current, older below
YesNo=Yes
; this message will be used for 'Yes / No' value in table
DocumentFrontPage=ProjectDetails,Warning,PeopleDetails,RevisionDetails,AddPurpose,RiskTable
WriteTableOfContent=Yes
; Write global Table Of Contents at the end of the file
TitleFlat=Yes
; so the titles will be all numerical and hierachical (without any big sections)
Landscape=Yes
; full body is to be written as Landscape

:Software Changes
\TableSoftwareChanges
:Cross References
: Reference and Related Documents
\TableDocuments
: Traceability Matrix
\TableTraceabilityMatrix
:Package Content
=[SoftwareVersion]
:Known issues
=[KnownIssues]

[Release]
Owner=DI
;Order=DI
; no Order= specified, so that the body of this [Release] document section will be written as plain text only
Name=Release Notes
Purpose=Present all software modifications introduced in the current release
PreparedBy=Arnaud Bouchez
ReviewedBy=
ApprovedBy=
Revision=1.8
RevisionDate=
RevisionDescription=Initial version
; Revision* multiple revision Table: ignored values are taken from current
DocumentFrontPage=ProjectDetails,Warning,PeopleDetails,RevisionDetails
WriteTableOfContent=Yes
; Write global Table Of Contents at the beginning of the file
TitleFlat=Yes
; so the titles will be all numerical and hierachical (without any big sections)

:Introduction
: Document Purpose
This @Release@ applies to the upgrade software for the {\i Synopse mORMot Framework}, implementing the main specifications detailed in the @DI@.
It describes the software requirements and bug corrections involved in this release.
: Software Version
This release updates the {\i mORMot Framework} software modules to the following versions.
=[SoftwareVersion]
: Software specifications
This release is compatible with the following software:
- Windows XP (or later) Operating system;
- {\i Delphi} 7 up to {\i Delphi} 2010.
:Release Notes
: New features
New features implemented in this release are listed below.
\TableNewFeatures
; all DI with Request=SCR #123 will be listed here
: Bug fixes
The following bugs reported in earlier versions have been fixed.
\TableBugFixes
; all DI with SCR #65,Module 2.0+Other Module 3.0,Low will be listed here
: Installation Instructions
The Installation of this release follows the steps detailed in the {\i mORMot Framework User Manual}, and did not change from previous version.
: Special Instructions for Use
As this release is mainly a bug fix, the instructions for use did not change, and the former {\i mORMot Framework User Manual} can be seen as a valid reference document for the User.
: Open Issues
In a general manner, the User has to follow strictly the procedures detailed in the documentation shipped with the {\i mORMot} Framework, and the corresponding software updates which may have been installed.
: Testing Status
The tests below were performed by strictly following the test protocols as stated by the @VV@, after having installed the release on a dedicated test computer.\line
\TableTests=...Date of test
All tests passed successfully. Therefore, this release should not prevent users from using features.
: Significant Faults
There are no significant remaining faults, which prevent using features.
See {\b Appendix B - Known Faults list} for the current open issues identified in the release, and are planned to be corrected.
\landscape
:Appendices
: Appendix A - Software Revision History
=[SoftwareHistory]
: Appendix B - Known Faults List
The following table lists the significant bugs that were found in this release:
=[KnownIssues]
: Appendix C - Release Documentation Audit
The following table is a partial list of related documentation, including the current revision numbers and revision dates for all documentation applicable to this release.
\TableDocuments=DI,SRS,Risk,SAD,SDD,VV,SCRS
: Appendix D - ISO 123456 cross reference
Here is a table of the implementation of the {\i ISO 123456} standard in all the documentation:
\TableImplements=ISO
At the beginning of the @RK@, @SRS@, @SAD@ and @SDD@, a dedicated table will list all {\i ISO 123456} requirements implemented in this document, with its associated page.

[SandBox]
Owner=DI
Name=SandBox
PreparedBy=Arnaud Bouchez

:Secure Communication using ECDHE
In addition to ECDSA digital signatures or ECDH-based content encryption, the {\i mORMot} framework offers a proprietary {\f1\fs20 @*ECDHE@} secure protocol for securing Client/Server communication. HTTPS/TLS should still be used with AJAX or third party endpoints. But this alternate protocol can be enabled, with both @140@ and @150@, between @*SOA@ nodes implemented with {\i mORMot} Delphi/FPC services. Advantages are easier deployment, better performance, reduced protocol complexity, and higher integration.
{\f1\fs20 SynEcc} implementation of {\f1\fs20 ECDHE} handshaking and key derivation is done in a single round trip, to avoid harmful triple handshakes, and reduce network latency. Both mutual authentication and server authentication are available, requiring a shared {\i public-key infrastructure} ({\f1\fs20 @*PKI@}) - provided e.g. by {\f1\fs20 TECCCertificateChain} - to validate exchanged certificates. Thanks to the use of ephemeral keys, handshaking features perfect forward security in its key derivation (used for encryption and message authentication). Messages transmission checks authentication, data integrity, and replay attacks, with hardware acceleration of the process, if available.
Thanks to the proven set of algorithms used, resulting security is comparable to the best TLS 1.2 configurations, without the overhead and complexity of this standard, and at very high speed.
: Mutual Authentication
To perform @*mutual authentication@, the prerequisite for each party is to have private keys ({\f1\fs20 dA} and {\f1\fs20 dB}) and public keys in certificates ({\f1\fs20 QCA} and {\f1\fs20 QCB}), hosted in a shared PKI system.
$Client (dA, QCA)                                            Server (dB, QCB)
When the client initiates the communication, it generates an ephemeral ({\f1\fs20 dE, QE}) ECC key pair (without certification), then send an identifier to the current algorithm {\f1\fs20 Algo}, a random value {\f1\fs20 RndA}, its own public key {\f1\fs20 QCA}, the ephemeral public key {\f1\fs20 QE}, concatenated and digitally signed with @*ECDSA@ using its private key {\f1\fs20 dA}.
$Client (dA, QCA)                                            Server (dB, QCB)
$
$  (dE, QE) = ECCMakeKey
$  Sign = ECDSASign(dA,sha-256(Algo|RndA|QCA|QE))
$
$    Algo|RndA|QCA|QE|Sign
$   ----------------------------------------------------------->
$
On the server side, the ECDSA signature is checked using {\f1\fs20 QCA} certificated public key, then an ephemeral ({\f1\fs20 dF, QF}) key pair is generated (without certification), and all information is sent back to the client, with a digital signature using Server private key {\f1\fs20 dB}. The signature is then validated using ECDSA on the client side, checking {\f1\fs20 QCB} certificate information.
$Client (dA, QCA)                                            Server (dB, QCB)
$
$                                                           ECDSAVerify(QCA, Sign)
$                                                           (dF, QF) = ECCMakeKey
$                                          Sign = ECDSASign(dB,sha-256(Algo|RndA|RndB|QCB|QF))
$
$                                    Algo|RndA|RndB|QCB|QF|Sign
$   <-----------------------------------------------------------
$
$  ECDSAVerify(QCB, Sign)
$
Now both ends can calculate shared secret keys {\f1\fs20 SA} and {\f1\fs20 SB}. Two session keys {\f1\fs20 kE} and {\f1\fs20 kM} are then derived using a {\f1\fs20 KDF} function (e.g. HMAC-SHA256). Subsequent {\f1\fs20 m1}, {\f1\fs20 m2}... messages will be encrypted using {\f1\fs20 kE} via an {\f1\fs20 EF} encryption function (e.g. AES256-CFB), and the current {\f1\fs20 IV} Initialization Vector, derived from the current {\f1\fs20 kM}. Finally, {\f1\fs20 kM} will authenticate them using a {\f1\fs20 MAC} function (e.g. HMAC-SHA256). {\f1\fs20 kM} value will increase as a CTR to maintain read and write sequence numbers on both sides, ensuring {\f1\fs20 IV} will change, and {\f1\fs20 MAC} won't suffer from replay attacks.
$Client (dA, QCA)                                            Server (dB, QCB)
$
$  SA = ECDH(dA,QF)                                         SA = ECDH(dF,QCA)
$  SB = ECDH(dE,QCB)                                        SB = ECDH(dB,QE)
$                  kE = KDF(SA|SB|RndA|RndB,"salt")
$                  kM = KDF(SA|SB|RndA|RndB,"hmac")
$
$   EF(kE,m1)|MAC(kM,EF(kE,m1))
$   +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++>
$  kM++
$                                     E(kE,m2)|MAC(kM,EF(kE,m2))
$   <+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
$                                                           kM++
$   EF(kE,m3)|MAC(kM,EF(kE,m3))
$   +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++>
$  kM++
$    ...
$
A typical {\f1\fs20 SynEcc} implementation may use, as algorithms:
- {\f1\fs20 KDF} = HMAC-SHA256 ("salt" and "hmac" values may be customized, but known on both sides);
- {\f1\fs20 EF} = AES128-CFB or any AES mode excluding ECB, potentially in 256-bit;
- {\f1\fs20 MAC} = HMAC-SHA256 (safest), HMAC-CRC256C (fast), or combined with {\f1\fs20 EF}.
By default, the {\f1\fs20 TECDHEProtocol} class will use {\f1\fs20 kdfHmacSha256} as {\f1\fs20 KDF}, and {\f1\fs20 efAesCrc128} (i.e. AES128-CFB with combined {\f1\fs20 EF} and {\f1\fs20 MAC}), for best performance (around 700MB/s messages process thanks to hardware accelerated @*AES-NI@ and SSE4.2 {\f1\fs20 @*crc32c@} instructions).
Note that encryption is not handled at this level, since all conservative protocol implementations do not enable compression, to avoid security exploit as occured for TLS with CRIME. It is up to the application layer to process the data using e.g. {\f1\fs20 deflate} or our {\f1\fs20 @*SynLZ@} algorithm.
: Unilateral Authentication
For server-side only authentication - as is most currently implemented in regular TLS/HTTPS communications, the handshaking process is slightly reduced:
$Client                                                      Server (dB, QCB)
$
$  (dE, QE) = ECCMakeKey
$
$    Algo|RndA|QE
$   ----------------------------------------------------------->
$
$                                          Sign = ECDSASign(dB,sha-256(Algo|RndA|RndB|QCB))
$
$                                       Algo|RndA|RndB|QCB|Sign
$   <-----------------------------------------------------------
$
$  ECDSAVerify(QCB, Sign)
$  S = ECDH(dE,QCB)                                         S = ECDH(dB,QE)
$                  kE = KDF(S|RndA|RndB,"salt")
$                  kM = KDF(S|RndA|RndB,"hmac")
$
$   EF(kE,m1)|MAC(kM,EF(kE,m1))
$   +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++>
$  kM++
$                                     E(kE,m2)|MAC(kM,EF(kE,m2))
$   <+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
$                                                           kM++
$    ...
$
In this case, the client party does not have any private/public key certification, and will compute an ephemeral {\f1\fs20 (dE, QE)} pair, which will be used using server's {\f1\fs20 CA}. The server has no mean of authenticating its client, but the connection is secured and private. The handshaking process will be slightly faster than with mutual authentication, since less ECC computing operations are performed (2 instead of 5 on the server side).
The protocol is also able to handle client-side only authentication, even if this scheme may not be very useful in practice.

