[People] ;Name=Function,Function details Arnaud Bouchez=Project Manager,Develop software and manage associated projects [Project] Name=Synopse mORMot Framework Company=Synopse ReleaseVersion= ReleaseDate= Manager=Arnaud Bouchez MainSection=DI ; so we first check for [DI] and [DILayout] NoRiskDisplay=Not implemented DestinationDir=D:\Documents\SynProject ; path to store all created .doc (not to be inside versioning tree) OldWordOpen=No ; if OldWordOpen=Yes, Conversion is made visible on the screen (compatible with some Word 2000 installations) DefLang=1033 Logo=logo.png ; this picture will be displayed top of every document front page HeaderColWidth=22,37,22,19 ; page header columns width, according to Manager=The Manager's name NoConfidential=Yes ; so that no "Confidential" text will appear in page footer - seems convenient for a GPL document ;) HeaderWithLogo=Yes ; custom page header with the synopse logo HtmlSideBar=Overview/Meet the mORMot:SOURCE,Download/How to install:TITL_113,API Reference/Units and classes:SIDE_MORMOT_FRAMEWORK,FAQ/Frequently Asked Questions:TITL_123,Forum/Get support:https://synopse.info/forum,TimeLine/Open Source:https://synopse.info/fossil/timeline,Blog/Latest News:http://blog.synopse.info,Donate/Adopt a mORMot!:https://synopse.info/fossil/wiki?name=HelpDonate,Licence Terms/Either MPL, LGPL or GPL:TITL_34 ; the sidebar first links, for html export {\b Document License} {\i Synopse mORMot Framework Documentation}.\line Copyright (C) 2008-2020 Arnaud Bouchez.\line Synopse Informatique - @https://synopse.info The {\i Synopse mORMot Framework Source Code} is licensed under GPL / LGPL / MPL licensing terms, free to be included in any application. ;This documentation has been generated using {\i Synopse SynProject} - @https://synopse.info/fossil/wiki?name=SynProject ;This document is a free document; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 3 of the License, or (at your option) any later version. The {\i Synopse mORMot Framework Documentation} is a free document, released under a GPL 3.0 License, distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. ;You should have received a copy of the GNU General Public License along with this documentation. If not, see @http://www.gnu.org/licenses {\b Trademark Notice} Rather than indicating every occurrence of a trademarked name as such, this document uses the names only in an editorial fashion and to the benefit of the trademark owner with no intention of infringement of the trademark. [Pictures] SmartCalculator.png=278x328 35%,Smart Mobile Studio Calculator Sample synfilevcl.png=1015x666 95%,User Interface generated using VCL components synfiletms.png=955x610 95%,User Interface generated using TMS components cartoon08.png=230x205 35%,Adopt a mORMot cartoon07.png=250x213 35%,Adopt a mORMot cartoon06.png=200x273 20%,Adopt a mORMot cartoon05.png=250x234 30%,Adopt a mORMot cartoon04.png=200x286 25%,Adopt a mORMot cartoon03.png=250x285 30%,Adopt a mORMot cartoon02.png=250x258 30%,Adopt a mORMot cartoon01.png=371x240 30%,Adopt a mORMot IamLost.png=300x283 30%,Meet the mORMot logo.png=200x47 25%,Synopse Logo ; here are stored the Pictures properties, as PictureFileName=WIDTHxHEIGHT PERCENT%,Caption [DI] Owner=DI Order=DI Name=Design Input Product Specifications ItemName=DI DisplayName=Design Input Purpose=Create high level description of software specifications PreparedBy=Arnaud Bouchez ReviewedBy= ApprovedBy= ; Revision* multiple revision Table: ignored values are taken from current, older below RevisionDescription=Initial Version RevisionDate= Revision=1.18 ; [DILayout] list the global DI outline (lines beginning with : are titles) ; [DI-*] details all items DefaultPreparedBy=Arnaud Bouchez ; all [DI-*] PreparedBy= default name DocumentFrontPage=ProjectDetails,Warning,PeopleDetails,RevisionDetails,AddPurpose ;SubDocFrontPage=Warning,PeopleDetails :System Specifications ; special lines begin with ; : for titles (no Name=Value pairs after a :title) ; :# Title for a later reference as @#@ (":1 Title" then @1@) ; - for a list item ; ! for pascal source ; !! for modified pascal source line ; & for c c++ source ; &! for modified c c++ source line ; # for c# source ; #! for modified c# source line ; $ for text file (fixed-width font) ; $! for modified text file line (fixed-width font) ; %filename.jpg [640x480 85%] for images jpg/bmp/png/emf - see [Pictures] ; %%FirmwareBoot for diagram images, i.e. not listed in [Pictures] but created with \graph FirmwareBoot ... ; |%30%40%30 then |Col 1|Col 2|Col 3 for every row, ending with |% for columns ; |%=-30%40%30 -> =:no indent -:no border ; =[SectionName] to inline the [SectionName] content at this place ; text can be formated as rtf (with \b \i { } e.g.) - each new text paragraph will be ended with \par ; {} can be used for a \par alone (void lines are just ignored) ; you can link to another item with @SectionName@ (@DI-4.1@ e.g) or @DocName@ (@SRS@) or @PeopleName@ (@A.Bouchez@) or either @%Picture.png@ ; you can embedd a picture within a table cell e.g., by using @=%picture.png@ - in this case, this is not a "button" ; internet links will be handled as hyperlink, with @http://synopse.info ; in the [SDD-*] sections, specify @Module\filename.pas@ for each file name, @!Module\filename.pas@ for filename modified or @!procedurename!Module\filename.pas@ in order to specify the procedure name. The corresponding units (and procedures) will be highlited in the [SAD] document. Just click on the button to use the Object Browser window. ; some special lines commands can be entered: ; \page to force a new page ; \landscape to change the page orientation to landscape ; \portrait to change the page orientation to portrait ; \footer blabla to change the footer text ; \Layout to add a list with all DILayout titles ; \LayoutPage idem + associated pages in the document ; \risk to add the Risk Assessment Scale table ; \Source (for [SAD] section) to add the list of the Source=.. modules ; \SourcePage idem + associated pages in the document ; \include filename.ext ext will be used to append !&#$ left ; \graph UniqueImageName [Title] then following lines either .dot normal text, or "\From Text\To Text[\Label between both]" - use F12 to have the dialog ; \TableSoftwareChanges or \TableTraceabilityMatrix for SCRS ; \TableNewFeatures or \TableBugFixes or \TableTests[=October 16, 2008] for Release Notes ; \TableDI=6.3.1.2,6.3.1.3 for a table with all the supplied Design Inputs ; \TableDocuments[=DI,SRS,SDD,SAD] for a table with the supplied document details ; \Implements TableImplementsName #.# [Description][\DocumentName] (\Implements ISO 4.3 Software safety classification) in the text - points to the current document, or the specified DocumentName ; \TableImplements=TableImplementsName (\TableImplements=ISO) to create the list, sorted by ascending #.# numbers, with description if any and corresponding document ; =[SectionName] to include this section content at the current place ; in the [Test-*] sections, special auto-defined columns can be used with |Actions[|Expected Results] - manual tables can be used as usual (with |%..) This document is intended to describe the Design Input Product Specifications. : Definitions {\b Added Value} - This level of achievement should be the target of the design team, because achieving this level of performance adds value to the product. However failure to achieve this level does not evoke additional management review. {\b Must Have} - This level of achievement must be reached in the final design output. Because of possible negative financial impacts, if this level of performance is not achieved, management review will be triggered. : Project Concept : Purpose and Scope This document focuses on the {\i Synopse mORMot Framework} library. The purpose of this @DI@ is to detail the marketing requirements/product specifications for the 1.18 release of the {\i Synopse mORMot Framework library}. The requirements and specifications found in this document are derived from customer market research, regulatory input and industry common practice. : Concept Statement It was observed that a true JSON and RESTful oriented Client-Server framework was missing in the {\i Delphi} programing environment. Latest versions of {\i Delphi} (i.e. {\i Delphi} 2010 and up) provide a JSON and RESTful mechanism named DataSnap (in the {\i Architect} or {\i Enterprise} editions), but such a feature could be implemented with previous versions of the {\i Delphi} compiler as well, with a more open architecture. This framework shall use a innovative ORM (Object-relational mapping) approach, based on the RTTI (Runtime Type Information) provided by the {\i Delphi} language. It shall expose Server data access and business services to Clients, using JSON over several communication protocols. After evaluation of most used database engines, the {\i SQLite3} engine was found out to be secure, fast, and perfectly adapted as a stand-alone database engine for this framework, able to access other remote database engines using its unique {\i Virtual Tables} mechanism. Together with this Client-Server data and business architecture, a set of User Interface components (especially Database Grid and Reporting system) are provided within the framework. The main approach of this framework is to avoid @*RAD@ in the development of projects. RAD has been proved to be a good candidate about prototyping, but is not the best approach for creating a robust and maintainable application. Best practices (as MVC, n-Tier or SOA) shall be used instead. : Expected Use Any application which need moderate database usage (up to some GB of data) with easy setup and administration, together with a secure @*ACID@ behavior in a Client-Server environment should consider using the {\i Synopse mORMot Framework}. : Requirement Exceptions This framework was developed in order to run mainly under any {\i Delphi} compiler, from version {\i Delphi} 6 to version {\i Delphi 10.3 Rio}. On the {\i server side}, it targets both {\i Win32} and {\i Win64} platforms (using the 64-bit compiler included in latest {\i Delphi} XE2 and up). For clients, in addition to those {\i Win32} / {\i Win64} platforms, you have cross-platform code generation abilities, for any {\i Delphi} or {\i @*FreePascal@} target (including {\i @*OSX@} and mobile {\i iOS} or {\i Android}), or AJAX / HTML5 clients via {\i @*Smart Mobile Studio@} - see @90@. =[License] \page :Software Design Input The Software @DI@ items follow these main divisions: \LayoutPage [DILayout] ; lines beginning with : will be titles for general DI layout - the 'DI-' chars are added before numbers listed below :Client Server ORM/SOA framework 2.1.1 2.1.1.1 2.1.1.2 2.1.2 2.1.3 2.1.4 2.1.5 :SQlite3 engine 2.2.1 2.2.2 2.2.3 :User interface 2.3 2.3.1 2.3.2 [DI-2.1.1] Risk=1,1,3,Arnaud Bouchez,Initial release Request=Initial release Ident=The framework shall be Client-Server oriented [DI-2.1.1.1] Risk=1,1,3,Arnaud Bouchez,Initial release Request=Initial release Ident=A RESTful mechanism shall be implemented [DI-2.1.1.2] Risk=1,1,3,Arnaud Bouchez,Initial release Request=Initial release Ident=Commmunication should be available directly in the same process memory, or remotly using Named Pipes, Windows messages or HTTP/1.1 protocols [DI-2.1.2] Risk=1,1,3,Arnaud Bouchez,Initial release Request=Initial release Ident=UTF-8 JSON format shall be used to communicate [DI-2.1.3] Risk=1,1,3,Arnaud Bouchez,Initial release Request=Initial release Ident=The framework shall use an innovative ORM (Object-relational mapping) approach, based on classes RTTI (Runtime Type Information) [DI-2.1.4] Risk=1,1,3,Arnaud Bouchez,Initial release Request=Initial release Ident=The framework shall provide some Cross-Cutting components [DI-2.1.5] Risk=1,1,3,Arnaud Bouchez,Initial release Request=Initial release Ident=The framework shall offer a complete SOA process [DI-2.2.1] Risk=1,1,3,Arnaud Bouchez,Initial release Request=Initial release Ident=The {\i SQLite3} engine shall be embedded to the framework [DI-2.2.2] Risk=1,1,3,Arnaud Bouchez,Initial release Request=Initial release Ident=The framework libraries, including all its {\i SQLite3} related features, shall be tested using Unitary testing [DI-2.2.3] Risk=1,1,3,Arnaud Bouchez,Initial release Request=Initial release Ident=The framework shall be able to access any external database, via OleDB, ODBC or direct access for Oracle (OCI) or SQLite3 (for external database files) [DI-2.3] Risk=1,1,3,Arnaud Bouchez,Initial release Request=Initial release Ident=User Interface and Report generation should be integrated [DI-2.3.1] Risk=1,1,3,Arnaud Bouchez,Initial release Request=Initial release Ident=An User Interface, with buttons and toolbars shall be easily being created from the code, with no RAD needed, using RTTI and data auto-description [DI-2.3.2] Risk=1,1,3,Arnaud Bouchez,Initial release Request=Initial release Ident=A reporting feature, with full preview and export as PDF or TXT files, shall be integrated [RK] ; self-owned Risk Asssessment document Owner=RK Order=RK ItemName=FMEA DisplayName=Design FMEA Name=Design FMEA File DocName=Design FMEA File ; this DocName will be used for generating .DOC filename (instead of ItemName) Purpose=List {\i Failure Modes and Effects Analysis} (FMEA) PreparedBy=Arnaud Bouchez ReviewedBy= ApprovedBy= Revision=1.18 RevisionDate= RevisionDescription=Initial Version ; Revision* multiple revision Table: ignored values are taken from current, older below DocumentFrontPage=ProjectDetails,Warning,PeopleDetails,RevisionDetails,AddPurpose WriteRisk=Yes ; Write Risk assessment table summary after every RK item WriteTableOfContent=Yes ; Write global Table Of Contents TableOfContentsAtTheBeginning=Yes ; if the Table of Contents must be at the beginning (default=No=at the end of the file) DocumentIndex=Pictures,Implements ISO=ISO 123456 requirements ; "Pictures" will create a table with all picture appearing in this document ; "Implements ISO=..." will create a table with all appearing "\Implements ISO 3.4" pages, with the specified item name :Introduction The @RK@ is a reference document used to list the {\i Failure Modes and Effects Analysis} (FMEA) identified for the {\i Synopse mORMot Framework} library. The "{\i Failure modes and effects analysis}" (FMEA) is a procedure in operations management for analysis of potential failure modes within a system for classification by severity or determination of the effect of failures on the system. {\i Failure modes} are any errors or defects in a process, design, or item, especially those that affect the customer, and can be potential or actual. {\i Effects analysis} refers to studying the consequences of those failures. In practice, a Risk Assessment team starts with a block diagram of a system. The team then considers what happens if each block of the diagram fails, and fills in a table in which failures are paired with their effects and an evaluation of the effects. The design of the system is then corrected, and the table adjusted until the system is known not to have unacceptable problems. This @RK@ lists most FMEA items identified as possible Software Failure for the {\i Synopse mORMot Framework}. : Risk Assessment In the following @RK@, a numerical Risk Assessment is given for every FMEA item, according to the {\i Risk Assessment Scale} table below. A summary explanation is indicated, together with the names of those who made each evaluation. \risk : Responsibilities - Synopse will try to correct any identified issue; - The Open Source community will create tickets in a public Tracker web site located at @https://synopse.info/fossil ; - Synopse work on the framework is distributed without any warranty, according to the chosen license terms; - This documentation is released under the GPL (GNU General Public License) terms, without any warranty of any kind. \page :FMEA : Fault Tree Here is the Fault Tree of the framework, displayed in a graphical way: \graph FTA mORMot Framework Fault Tree \mORMot Framework\Framework Architecture \Framework Architecture\Invalid Concurent Access \Invalid Concurent Access\Database corruption \Invalid Concurent Access\Wrong Client-Server synchro \Wrong Client-Server synchro\Enduser problems \Framework Architecture\Main Server Crashed \Framework Architecture\Security issue \Security issue\Enduser problems \Security issue\Database corruption \Main Server Crashed\Database corruption \Database corruption\Enduser problems \mORMot Framework\User Interface \User Interface\Security issue \User Interface\Inconsistent Layout \Inconsistent Layout\Timeout problems \Inconsistent Layout\Incorrect User action \Incorrect User action\Enduser problems \User Interface\Function not working \Function not working\Timeout problems \ [SRS] Owner=DI Order=SRS ; Owner: [SRS-*] -> * reference; Order=SRS -> [SRS-*] have sub items ;Refers=RK ; Refers will add all [SRS-RK-*] items to the list, after the DI Name=Software Requirements Specifications ItemName=SWRS Purpose=Interpret design inputs and specify software design features PreparedBy=Arnaud Bouchez ReviewedBy= ApprovedBy= Revision=1.18 RevisionDate= RevisionDescription=Initial Version ; Revision* multiple revision Table: ignored values are taken from current, older below ;PreparedBy=.. ignored values are taken from current ; [SRS-*] sections describe each item ([DI] items + other items) ; [SRS-*] are displayed as they appear in the file DocumentFrontPage=ProjectDetails,Warning,PeopleDetails,RevisionDetails,AddPurpose WriteRisk=Yes ; Write Risk assessment table summary after every DI WriteTableOfContent=Yes ; Write global Table Of Contents at the end of the file TableOfContentsAtTheBeginning=Yes ; if the Table of Contents must be at the beginning (default=No=at the end of the file) DocumentIndex=Pictures,Implements ISO=ISO 123456 requirements :Introduction : Documentation overview The whole Software documentation process follows the typical steps of this diagram: \graph FMEADI Design Inputs, FMEA and Risk Specifications \User¤Requirements\Design Inputs¤(DI)\define \Regulatory¤Requirements\Design Inputs¤(DI) \Design Inputs¤(DI)\Specifications¤(SWRS)\are specified by \System-wide¤Risk Assessment\SW FMEA¤(RK)\defines \SW FMEA¤(RK)\Specifications¤(SWRS) \Specifications¤(SWRS)\Architecture + Design¤(SAD+SDD)\is implemented by \Architecture + Design¤(SAD+SDD)\Test + Documentation\is associated to \Test + Documentation\Specifications¤(SWRS)\refers to \ : Purpose This @SRS@ applies to the first public release of the {\i Synopse mORMot Framework}. It describes the software implementation of each design input as specified by the @DI@. The sections of this document follow the @DI@ divisions: \LayoutPage ;Then all items created from the @RK@ are listed: ;\referspage For each Design Input item, the corresponding justification is specified, between parenthesis (SCR #65, e.g.). Every @SRS@ item is named about the corresponding @DI@ item, or, in case the initial {\i Design Input} is too large and must be divided into some {\i SWRS} more precise items, an unique name is proposed. : Risk Assessment The Risk assessment indicated below was evaluated as a team work, based on the software solution proposed. In the following @SRS@, a numerical Risk Assessment is given for every Design Input item, according to the {\i Risk Assessment Scale} table below. A summary explanation is indicated, together with the names of those who made each evaluation. \risk : Responsibilities - Synopse will try to correct any identified issue; - The Open Source community will create tickets in a public Tracker web site located at @https://synopse.info/fossil ; - Synopse work on the framework is distributed without any warranty, according to the chosen license terms; - This documentation is released under the GPL (GNU General Public License) terms, without any warranty of any kind. [SRS-DI-2.1.1] ; DI-2.1.1 - The framework shall be Client-Server oriented ShortName=Client-Server framework Client–Server model of computing is a distributed application structure that partitions tasks or workloads between service providers, called servers, and service requesters, called clients. Often clients and servers communicate over a computer network on separate hardware, but both client and server may reside in the same system. A server machine is a host that is running one or more server programs which share its resources with clients. A client does not share any of its resources, but requests a server's content or service function. Clients therefore initiate communication sessions with servers which await (listen for) incoming requests. The {\i Synopse mORMot Framework} shall implement such a Client-Server model by a set of dedicated classes, over various communication protocols, but in an unified way. Application shall easily change the protocol used, just by adjusting the class type used in the client code. By design, the only requirement is that protocols and associated parameters are expected to match between the Client and the Server. [SRS-DI-2.1.1.1] ; DI-2.1.1.1 - A RESTful mechanism shall be implemented ShortName=RESTful framework REST-style architectures consist of clients and servers, as was stated in @SRS-DI-2.1.1@. Clients initiate requests to servers; servers process requests and return appropriate responses. Requests and responses are built around the transfer of "representations" of "resources". A resource can be essentially any coherent and meaningful concept that may be addressed. A representation of a resource is typically a document that captures the current or intended state of a resource. In the {\i Synopse mORMot Framework}, so called "resources" are individual records of the underlying database, or list of individual fields values extracted from these databases, by a SQL-like query statement. [SRS-DI-2.1.1.2] ; DI-2.1.1.2 - Commmunication should be available directly in the same process memory, or remotly using Named Pipes, Windows messages or HTTP/1.1 protocols ShortName=Communication via diverse protocols In computing and telecommunications, a protocol or communications protocol is a formal description of message formats and the rules for exchanging those messages. The {\i Synopse mORMot Framework} shall support the following protocols for remote access, according to the Client-Server architecture defined in @SRS-DI-2.1.1@: - Direct in-process communication; - Using Windows Messages; - Using Named pipe; - Using HTTP/1.1 over TCP/IP. [SRS-DI-2.1.1.2.1] Parent=DI-2.1.1.2 Ident=Client-Server Direct communication shall be available inside the same process ShortName=In-Process communication [SRS-DI-2.1.1.2.2] Parent=DI-2.1.1.2 Ident=Client-Server Named Pipe communication shall be made available by some dedicated classes ShortName=Named Pipe protocol [SRS-DI-2.1.1.2.3] Parent=DI-2.1.1.2 Ident=Client-Server Windows Messages communication shall be made available by some dedicated classes ShortName=Windows Messages protocol [SRS-DI-2.1.1.2.4] Parent=DI-2.1.1.2 Ident=Client-Server HTTP/1.1 over TCP/IP protocol communication shall be made available by some dedicated classes, and ready to be accessed from outside any {\i Delphi} Client (e.g. the implement should be AJAX ready) ShortName=HTTP/1.1 protocol [SRS-DI-2.1.2] ; DI-2.1.2 - UTF-8 JSON format shall be used to communicate JSON, as defined in the @SAD@, is used in the {\i Synopse mORMot Framework} for all Client-Server communication. JSON (an acronym for {\i JavaScript} Object Notation) is a lightweight text-based open standard designed for human-readable data interchange. Despite its relationship to {\i JavaScript}, it is language-independent, with parsers available for virtually every programming language. JSON shall be used in the framework for returning individual database record content, in a disposition which could make it compatible with direct {\i JavaScript} interpretation (i.e. easily creating {\i JavaScript} object from JSON content, in order to facilitate AJAX application development). From the Client to the Server, record content is also JSON-encoded, in order to be easily interpreted by the Server, which will convert the supplied field values into proper SQL content, ready to be inserted to the underlying database. JSON should be used also within the transmission of request rows of data. It therefore provide an easy way of data formating between the Client and the Server. The {\i Synopse mORMot Framework} shall use UTF-8 encoding for the character transmission inside its JSON content. UTF-8 (8-bit Unicode Transformation Format) is a variable-length character encoding for Unicode. UTF-8 encodes each character (code point) in 1 to 4 octets (8-bit bytes). The first 128 characters of the Unicode character set (which correspond directly to the ASCII) use a single octet with the same binary value as in ASCII. Therefore, UTF-8 can encode any Unicode character, avoiding the need to figure out and set a "code page" or otherwise indicate what character set is in use, and allowing output in multiple languages at the same time. For many languages there has been more than one single-byte encoding in usage, so even knowing the language was insufficient information to display it correctly. [SRS-DI-2.1.3] ; DI-2.1.3 - The framework shall use an innovative ORM (Object-relational mapping) approach, based on classes RTTI (Runtime Type Information) ORM, as defined in the @SAD@, is used in the {\i Synopse mORMot Framework} for accessing data record fields directly from {\i Delphi} Code. Object-relational mapping (ORM, O/RM, and O/R mapping) is a programming technique for converting data between incompatible type systems in relational databases and object-oriented programming languages. This creates, in effect, a "virtual object database" that can be used from within the {\i Delphi} programming language. The {\f1\fs20 published} properties of classes inheriting from a new generic type named {\f1\fs20 TSQLRecord} are used to define the field properties of the data. Accessing database records (for reading or update) shall be made by using these classes properties, and some dedicated Client-side methods. [SRS-DI-2.1.4] ; DI-2.1.4 - The framework shall provide some Cross-Cutting components {\i Cross-Cutting infrastructure layers} shall be made available for handling data filtering and validation, security, session, cache, logging and testing (framework uses test-driven approach and features stubbing and mocking). All crosscutting scenarios are coupled, so you benefit of consisting APIs and documentation, a lot of code-reuse, JSON/RESTful orientation from the ground up. [SRS-DI-2.1.5] ; DI-2.1.5 - The framework shall offer a complete SOA process In order to follow a {\i Service Oriented Architecture} design, your application's business logic can be implemented in several ways using {\i mORMot}: - Via some {\f1\fs20 @*TSQLRecord@} inherited classes, inserted into the database {\i model}, and accessible via some @*REST@ful URI - this is implemented by our @*ORM@ architecture - see @SRS-DI-2.1.3@; - By some RESTful @**service@s, implemented in the Server as {\i published methods}, and consumed in the Client via native {\i Delphi} methods; - Defining some RESTful {\i service @*contract@s} as standard {\i Delphi} {\f1\fs20 interface}, and then run it seamlesly on both client and client sides. [SRS-DI-2.2.1] ; DI-2.2.1 - The SQLite3 engine shall be embedded to the framework The {\i SQLite3} database engine is used in the {\i Synopse mORMot Framework} as its kernel database engine. {\i SQLite3} is an ACID-compliant embedded relational database management system contained in a C programming library. This library shall be linked statically to the {\i Synopse mORMot Framework}, or using official external {\f1\fs20 sqlite3.dll} distribution, and interact directly from the {\i Delphi} application process. The {\i Synopse mORMot Framework} shall enhance the standard {\i SQLite3} database engine by introducing some new features stated in the @SAD@, related to the Client-Server purpose or the framework - see @SRS-DI-2.1.1@. [SRS-DI-2.2.2] ; DI-2.2.2 - The framework libraries, including all its {\i SQLite3} related features, shall be tested using Unitary testing The {\i Synopse mORMot Framework} shall use all integrated Unitary testing features provided by a common testing framework integrated to all Synopse products. This testing shall be defined by classes, in which individual published methods define the actual testing of most framework features. All testing shall be run at once, for example before any software release, or after any modification to the framework code, in order to avoid most regression bug. [SRS-DI-2.2.3] ; DI-2.2.3 - The framework shall be able to access any external database, via OleDB or direct access for Oracle (OCI) or SQLite3 (for external database files) The following external database providers shall be made available to the framework ORM: - Any {\i OleDB} provider; - Any {\i ODBC} provider; - Any {\f1\fs20 TDataset} / {\f1\fs20 DB.pas} provider; - {\i Oracle} database, via direct OCI client access; - {\i SQlite3} database engine. A dedicated set of classes shall implement access, together with some advanced syntaxic sugar, like fast late-binding, or advanced ORM mechanism, like Virtual Tables. [SRS-DI-2.3] ; DI-2.3 - User Interface and Report generation should be integrated The {\i Synopse mORMot Framework} shall provide User Interface and Report generation from code. Such a ribbon-oriented interface shall be made available, in a per-table approach, and associated reports. Here is a sample of screen content, using proprietary TMS components: %synfiletms.png And here is the same application compiled using only VCL components, available from {\i Delphi} 6 up to {\i Delphi 10.3 Rio}: %synfilevcl.png [SRS-DI-2.3.1] ; DI-2.3.1 - An User Interface, with buttons and toolbars shall be easily being created from the code, with no RAD needed, using RTTI and data auto-description The {\i Synopse mORMot Framework} shall provide some User-Interface dedicated units, allowing database grid display, on screen tool-bar creation (by an internal system of actions using {\i Delphi} RTTI - see the corresponding paragraph in the @SAD@), and integrated reporting - see @SRS-DI-2.3.2@. No RAD approach is to be provided: the Client application User Interface will be designed not by putting components on the IDE screen, but directly from code. [SRS-DI-2.3.1.1] Parent=DI-2.3.1 Ident=A Database Grid shall be made available to provide data browsing in the Client Application - it shall handle easy browsing, by column resizing and sorting, on the fly customization of the cell content ShortName=Database Grid Display [SRS-DI-2.3.1.2] Parent=DI-2.3.1 Ident=Toolbars shall be able to be created from code, using RTTI and enumerations types for defining the action ShortName=RTTI generated Toolbars [SRS-DI-2.3.1.3] Parent=DI-2.3.1 Ident=Internationalization (i18n) of the whole User Interface shall be made available by defined some external text files: {\i Delphi} resourcestring shall be translatable on the fly, custom window dialogs automaticaly translated before their display, and User Interface generated from RTTI should be included in this i18n mechanism ShortName=Automated i18n [SRS-DI-2.3.2] ; DI-2.3.2 - A reporting feature, with full preview and export as PDF or TXT files, shall be integrated The {\i Synopse mORMot Framework} shall provide a reporting feature, which could be used stand-alone, or linked to its database mechanism. Reports shall not be created using a RAD approach (e.g. defining bands and fields with the mouse on the IDE), but shall be defined from code, by using some dedicated methods, adding text, tables or pictures to the report. Therefore, any kind of report shall be generated. This reports shall be previewed on screen, and exported as PDF or TXT on request. [Risk] Owner=SRS Order=SRS ; Owner: [Risk-*] -> * reference; Order=SRS -> [Test-*] have no sub items Name=Software Implementation Risk Assessment DocName=Risk Assessment ; this DocName will be used for generating .DOC filename (instead of ItemName) Purpose=Perform a risk assessment of the SWRS implementation PreparedBy=Arnaud Bouchez ReviewedBy= ApprovedBy= Revision=1.18 RevisionDate= RevisionDescription=Initial Version ; Revision* multiple revision Table: ignored values are taken from current, older below ; rev.1 summary values are in [DI] Risk=Severity,Probability,Occurence,Comment ; [Risk-*] sections contain rev.2 details for each SRS, [Risk-SER-03] or [Risk-4.5.1] e.g. DocumentFrontPage=ProjectDetails,Warning,PeopleDetails,RevisionDetails,AddPurpose,RiskTable [SAD] Owner=SRS Order=SRS ; Owner: [SAD-*] -> * reference; Order=SRS -> [SAD-*] have no sub items Name=Software Architecture Design Purpose=Describe the implications of each software requirement specification on all the affected software modules PreparedBy=Arnaud Bouchez ReviewedBy= ApprovedBy= Revision=1.18 RevisionDate= RevisionDescription=Initial Version ; Revision* multiple revision Table: ignored values are taken from current, older below ; [SAD-*] sections contain details for each SRS, [SAD-SER-03] or [SAD-DI-4.10.6] e.g. ; [SAD-*] are displayed as they appear in the [SRS-*] sections DocumentFrontPage=ProjectDetails,Warning,PeopleDetails,RevisionDetails,AddPurpose Source=Main,SynFile ; presence of Source=.. adds global description for each SourceFile=.. and for @Module\filename.pas@ for the project (TProject.Parse: array of TSection - [SAD-Module] will get files from @Module\name.pas@, e.g.) SourceSDD=SDD ; name of the [SDD] section to be parsed for modified files as @!Module\bidule.pas@ DefaultPath=D:\Dev ; global default directory to trim in the documentation, and used by default in [SAD-module] SourcePath=.. Directives=INCLUDE_FTS3 ; optional global conditional define when parsing the source WriteTableOfContent=Yes ; Write global Table Of Contents of the file TableOfContentsAtTheBeginning=Yes ; if the Table of Contents must be at the beginning (default=No=at the end of the file) DocumentIndex=Pictures,Source,Index ; "Source" will create a table with all @source.code@ files within this document ; "Index" an index of all @*keyword@, Picture of all %pictures WithAllfields=Yes ; write all field properties of all object and classes: document is huge, but contains all available architecture intel :Foreword The whole Software documentation process follows the typical steps of this diagram: %%FMEADI : Purpose This @SAD@ applies to the 1.18 release of the {\i Synopse mORMot Framework} library. After a deep presentation of the framework architecture and main features, each source code unit is detailed, with clear diagrams and tables showing the dependencies between the units, and the class hierarchy of the objects implemented within. The {\i SynFile} main demo is presented on its own, and can be used as a general {\i User Guide} of its basic ORM features and User Interface generation - see @50@. At the end of this document, @SRS@ items are linked directly to the class or function involved with the @SDD@, from the source code. : Responsibilities - Support is available in the project forum - @https://synopse.info/forum - from the {\i mORMot} Open Source community; - Tickets can be created in a public Tracker web site located at @https://synopse.info/fossil , which publishes also the latest version of the project source code; - Synopse can provide additional support, expertise or enhancements, on request; - Synopse work on the framework is distributed without any warranty, according to the chosen license terms - see @34@; - This documentation is released under the GPL ({\i GNU @*General Public License@}) terms, without any warranty of any kind. =[GPL] [SAD-Source] ; this Section body is added as the introduction of the document first part TitleOffset=0 DisplayName=mORMot Framework Overview :Synopse mORMot Overview %IamLost.png {\i Synopse mORMot} is an Open Source @*Client-Server@ @*ORM@ @*SOA@ @*MVC@ framework for {\i Delphi} 6 up to {\i Delphi 10.3 Rio} and @*FPC@, targeting {\i Win/@*Linux@} for the server, and any platform for clients (including mobile or AJAX). The main features of {\i mORMot} are therefore: - {\i ORM/ODM}: objects persistence on almost any database (SQL or NoSQL); - {\i SOA}: organize your business logic into @*REST@ services; - {\i Clients}: consume your data or services from any platform, via ORM classes or SOA interfaces; - {\i Web MVC}: publish your ORM/SOA process as responsive @*Web Application@s. With local or remote access, via an auto-configuring Client-Server @*REST@ design. \graph mORMotDesignORMSOA General mORMot architecture subgraph cluster_0 { "SQLDB"; label="SQL Databases"; } subgraph cluster_1 { "NoSQLDB"; label="NoSQL Databases"; } subgraph cluster_2 { "Services"; label="Services"; } subgraph cluster_3 { \SQLDB\ORM \NoSQLDB\ODM \REST Server\MVC/MVVM¤Web Server \ORM\REST Server \ODM\REST Server \Services\SOA \SOA\REST Server label=" mORMot\nServer"; } \REST Server\Stand Alone¤Application subgraph cluster_4 { label="REST Clients"; \REST Server\... any \REST Server\AJAX \REST Server\Mobile \REST Server\Delphi } subgraph cluster_5 { label=" Web Clients"; \MVC/MVVM¤Web Server\Desktop \MVC/MVVM¤Web Server\ Mobile } subgraph cluster_6 { label=" Featuring"; "Featured"; } =SQLDB=SQLite3 - Firebird - NexusDB\nPostgreSQL - MySQL - DB2\nMS SQL - Oracle - Informix =NoSQLDB=MongoDB\nIn-Memory\nFiles =Services=Method-based Services\nInterface-based Services\nAsynchronous (Push) Services\nRemote (Saas) Services =Featured=User Management - Security & Rights - Sessions - Replication¤Unit Testing - Mocks/Stubs - Logging - Performance - Profiling¤http.sys - WebSockets - MultiCore - Templates (MVC) ¤JSON - JavaScript Engine - Reporting - PDF - UI \ {\i mORMot} offers all features needed for building any kind of modern software project, with state-of-the-art integrated software components, designed for both completeness and complementarity, offering {\i @*convention over configuration@} solutions, and implemented for speed and efficiency. For {\i storing some data}, you define a {\f1\fs20 class}, and the framework will take care of everything: routing, JSON marshalling, table creation, SQL generation, validation. For {\i creating a service}, you define an {\f1\fs20 interface} and a {\f1\fs20 class}, and you are done. Of course, the same ORM/ODM or SOA methods will run on both server and client sides: code once, use everywhere! For {\i building a MVC web site}, write a Controller class in Delphi, then some HTML Views using {\i @*Mustache@} templates, leveraging the same ORM/ODM or SOA methods as Model. If you need a HTTP server, a proxy @*redirection@, master/slave @*replication@, @*publish-subscribe@, a test, a mock, add security, define users or manage rights, a script engine, a report, User Interface, switch to XML format or publish HTML dynamic pages - just pick up the right {\f1\fs20 class} or method. If you need a tool or feature, it is probably already there, waiting for you to use it. The table content of this document makes it clear: this is no ordinary piece of software. The {\i mORMot} framework provides an Open Source {\i self-sufficient set of units} (even {\i Delphi} starter edition is enough) for creating any {\i Multi-@*tier@} application, up to the most complex {\i @*Domain-Driven@} design - see @54@: - {\i Presentation layer} featuring @*MVC@ UI generation with @*i18n@ and reporting for rich {\i Delphi} clients, {\i @*Mustache@}-based templates for web views - see @108@ - or rich @*AJAX@ clients; - {\i Application layer} implementing Service Oriented Architecture via {\f1\fs20 interface}-based services (like @*WCF@) and Client-Server ORM - following a @*REST@ful model using @*JSON@ over several communication protocols (e.g. @*HTTP@/1.1 and @*HTTPS@); - {\i Domain Model layer} handling all the needed business logic in plain {\i Delphi} objects, including high-level managed types like @*dynamic array@s or records for {\i Value Objects}, dedicated classes for {\i Entities} or {\i Aggregates}, and {\f1\fs20 variant} storage with late-binding for dynamic documents - your business logic may also be completed in {\i JavaScript} on the server side as stated @79@; - {\i Data persistence infrastructure layer} with ORM persistence on direct @*Oracle@, @*MS SQL@, @*OleDB@, @*ODBC@, @*Zeos@ connection or any {\f1\fs20 DB.pas} provider (e.g. @*NexusDB@, @*DBExpress@, @*FireDAC@, @*AnyDAC@, @*UniDAC@...), with a powerful @*SQLite3@ kernel, and direct @*SQL@ access if needed - including SQL auto-generation for {\i SQLite3, Oracle, @*Jet/MSAccess@, MS SQL, @*Firebird@, @*DB2@, @*PostgreSQL@, @*MySQL@, @*Informix@} and {\i NexusDB} - the ORM is also able to use @*NoSQL@ engines via a native {\i @*MongoDB@} connection, for ODM persistence; - {\i Cross-Cutting infrastructure layers} for handling data filtering and validation, @*security@, @*session@, @*cache@, logging and @*test@ing (framework uses @*test-driven@ approach and features @*stub@bing and @*mock@ing). If you do not know some of those concepts, don't worry: this document will detail them - see @40@. With {\i mORMot}, {\i ORM} is not used only for data persistence of objects in databases (like in other implementations), but as part of a global n-@*Tier@, Service Oriented Architecture (SOA), ready to implement {\i Domain-Driven} solutions.\line {\i mORMot} is not another ORM on which a transmission layer has been added, like almost everything existing in Delphi, C# or Java: this is a full Client-Server ORM/SOA from the ground up. This really makes the difference. The business logic of your applications will be easily exposed as {\i Services}, and will be accessible from light clients (written in {\i Delphi} or any other mean, including AJAX). The framework Core is non-visual: it provides only a set of classes to be used from code. But you have also some UI units available (including screen auto-creation, reporting and ribbon GUI), and you can use it from any RAD, web, or AJAX clients. No dependency is needed at the client side (no DB driver, or third-party runtime): it is able to connect via standard HTTP or HTTPS, even through a corporate proxy or a VPN. Rich {\i Delphi} clients can be deployed just by copying and running a @*stand-alone@ small executable, with no installation process. Client authentication is performed via several secure methods, and communication can be encrypted via HTTS or with a proprietary SHA/@*AES@-256 algorithm. SOA endpoints are configured automatically for each published interface on both server and client sides, and creating a load-balancing proxy is a matter of one method call. Changing from one database engine to another is just a matter of one line of code; full audit-trail history is available, if needed, to track all changes of any class persisted by the ORM/ODM. Cross-platform clients can be easily created, as {\i Win32} and {\i Win64} executables of course, but also for any platform supported by the {\i Delphi} compiler (including {\i Mac @*OSX@}, {\i @*iPhone@}/{\i @*iPad@} and {\i @*Android@}), or by {\i @*FreePascal@} / {\i @*Lazarus@}. AJAX applications can easily be created via {\i @*Smart Mobile Studio@}, as will any mobile operating system be accessible as an HTML5 web rich client or stand-alone {\i @*PhoneGap@} application, ready to be added to the {\i Windows}, {\i Apple} or {\i Google} store. See @86@ for how {\i mORMot} client code generation leverages all platforms. Speed and scalability has been implemented from the ground up - see @59@: a genuine optimized multi-threaded core let a single server handle more than 50,000 concurrent clients, faster than {\i DataSnap}, {\f1\fs20 WCF} or {\f1\fs20 node.js}, and our rich SOA design is able to implement both vertical and horizontal scalable @*hosting@, using recognized enterprise-level SQL or NoSQL databases for storage. In short, with {\i mORMot}, your ROI is maximized. \page :41 Client-Server ORM/SOA framework The {\i Synopse mORMot framework} implements a Client-Server @*REST@ful architecture, trying to follow some MVC, N-@*Tier@, ORM, SOA best-practice patterns - see @40@. Several clients, can access to the same remote or local server, using diverse communication protocols: \graph mORMotDesign1 General mORMot architecture - Client / Server \Internet (VPN)\Local Network node [shape=box]; \Local Network\Server subgraph cluster_0 { "Client 1\n(Delphi)"; label="PC 1"; } subgraph cluster_1 { "Client 2\n(AJAX)"; label="PC 2"; } subgraph cluster_2 { \Client 4¤(Delphi)\Server \Server\Client 4¤(Delphi)\JSON + REST¤named pipe¤connection label="PC Server"; } subgraph cluster_3 { "Client n\n(Delphi)"; label="PC n"; } subgraph cluster_4 { "Client 3\n(Delphi)"; label="PC 3"; } \Client 1¤(Delphi)\Local Network\JSON + REST¤over HTTP/1.1 \Client 2¤(AJAX)\Internet (VPN)\JSON + REST¤over HTTP/1.1 \Client 3¤(Delphi)\Local Network \Client n¤(Delphi)\Internet (VPN) \ Or the application can be @*stand-alone@: \graph mORMotDesign2 General mORMot architecture - Stand-alone application rankdir=LR; node [shape=box]; subgraph cluster { \Client\Server\direct ¤access \Server\Client label="Stand-Alone application"; } \ Switch from this embedded architecture to the Client-Server one is just a matter of how {\i mORMot} classes are initialized. For instance, the very same executable can even be running as a stand-alone application, a server, or a client, depending on some run-time parameters! \page : Highlights At first, some points can be highlighted, which make this framework distinct to other available solutions: - @*Client-Server@ orientation, with optimized request caching and intelligent update over a @*REST@ful architecture - but can be used in stand-alone applications; - No @*RAD@ components, but true @*ORM@ and @*SOA@ approach; - Multi-@*Tier@ architecture, with integrated @*Business rules@ as fast ORM-based classes and {\i @*Domain-Driven@} design; - {\i Service-Oriented-Architecture} model, using custom RESTful JSON services - you can send as JSON any {\f1\fs20 TStrings, TCollection, TPersistent} or {\f1\fs20 TObject} (via registration of a custom serializer) instance, or even a {\i dynamic array}, or any record content, with integrated JSON @*serialization@, via an @*interface@-based contract shared on both client and server sides; - Truly RESTful @*authentication@ with a dual @*security@ model (session + per-query); - Very fast @*JSON@ producer and parser, with caching at SQL level; - Fast a configuration-less @*HTTP@ / @*HTTPS@ server using {\i @*http.sys@} kernel-mode server - but may communicate via named pipes, Windows Messages or in-process as lighter alternatives; - Using {\i @*SQLite3@} as its kernel, but able to connect to any other database (via @*OleDB@ / @*ODBC@ / @*Zeos@ or direct client library access e.g. for @*Oracle@) - the {\f1\fs20 SynDB.pas} classes are self-sufficient, and do not depend on the {\i Delphi} {\f1\fs20 DB.pas} unit nor any third-party (so even the {\i Delphi} Starter edition is enough) - but the {\f1\fs20 SynDBDataset} unit is also available to access any {\f1\fs20 DB.pas} based solution (e.g. @*NexusDB@, @*DBExpress@, @*FireDAC@, @*AnyDAC@, @*UniDAC@ or even the @*BDE@...); - RESTful ORM access to a @*NoSQL@ database engine like {\i @*MongoDB@} with the same code base; - Ability to use @*SQL@ and RESTful requests over multiple databases at once (thanks to {\i SQLite3} unique @*Virtual Table@s mechanism); - Full @*Text Search@ engine included, with enhanced Google-like ranking algorithm; - Server-side @*JavaScript@ engine, for defining your business intelligence; - Direct User Interface generation: grids are created on the fly, together with a modern Ribbon ('Office 2007'-like) screen layout - the code just has to define actions, and assign them to the tables, in order to construct the whole interface from a few lines of code, without any IDE usage; - Integrated @*Report@ing system, which could serve complex @*PDF@ reports from your application; - Designed to be as fast as possible (asm used when needed, buffered reading and writing avoid most memory consumption, multi-thread ready architecture...) so benchmarks sound impressive when compared to other solutions - see @59@; - More than 1800 pages of documentation; - {\i Delphi}, {\i FreePascal}, mobile and @*AJAX@ clients can share the same server, and ORM/SOA client access code can be generated on request for any kind of application - see @86@; - Full source code provided - so you can enhance it to fulfill any need; - Works from {\i Delphi} 6 up to {\i Delphi 10.3 Rio} and FPC 2.6.4/2.7.1/3.x, truly Unicode (uses @*UTF-8@ encoding in its kernel, just like JSON), with any version of {\i Delphi} (no need to upgrade your IDE). \page : Benefits As you can see from the previous section, {\i mORMot} provides a comprehensive set of features that can help you to manage your crosscutting concerns though a reusable set of components and core functionality. %IamLost.png Of course, like many developers, you may suffer from the well-known NIH ("Not Invented Here") syndrome. On the other side, it is a commonly accepted fact that the use of standard and proven code libraries and components can save development time, minimize costs, reduce the use of precious test resources, and decrease the overall maintenance effort. Benefits of {\i mORMot} are therefore: - @*KISS@ {\i convention over configuration} design: you have all needed features at hand, but with only one way of doing it - less configuration and less confusion for the developer and its customers; - Pascal oriented: implementation is not following existing Java or C# patterns (with generics (ab)use, variable syntaxes and black-box approach), but try to unleash the object pascal genius; - Integrated: all crosscutting scenarios are coupled, so you benefit of consisting APIs and documentation, a lot of code-reuse, JSON/RESTful orientation from the ground up; - Tested: most of the framework is @*test-driven@, and all regression tests are provided, including system-wide integration tests; - Do-not-reinvent-the-wheel, since we did it for you: it is now time to focus on your business; - Open Source, documented and maintained: project is developed since years, with some active members - {\i mORMot} won't leave you soon! \page :66 Legacy code and existing projects Even if {\i mORMot} will be more easily used in a project designed from scratch, it fits very well the purpose of evolving any existing {\i Delphi} project, or creating the server side part of an AJAX application.\line One benefit of such a framework is to facilitate the transition from a traditional @*Client-Server@ architecture to a N-@*Tier@ layered pattern. Due to its modular design, you can integrate some framework bricks to your existing application: - You may add logging to your code - see @16@, to track unresolved issues, and add customer-side performance profiling; - Use low-level classes like {\f1\fs20 record} or {\i dynamic array} wrappers - see @48@, or our dynamic document storage via {\f1\fs20 variant} - see @80@, including @*JSON@ or binary persistence; - You can use the direct DB layers, including the {\f1\fs20 @*TQuery@} emulation class - see @133@ - to replace some @**BDE@ queries, or introduce nice unique features like direct database access or {\i @*array bind@ing} for very fast data insertion - see @27@, or switch to a @*NoSQL@ database - see @83@; - Reports could benefit of the {\f1\fs20 mORMotReport.pas} code-based system, which is very easy to use even on the server side (serving @*PDF@ files), when your business logic heavily relies on objects, not direct DB - see @67@; - HTTP requests may be made available using Client-Server services via methods - see @49@, e.g. for rendering HTML pages generated on the fly with {\i @*Mustache@} templates- see @81@, pictures or @*PDF@ reports; - You can little by little move your logic out of the client side code into some server services defined via interfaces, without the overhead of SOAP or WCF - see @63@; migration to @*SOA@ is the main benefit of {\i mORMot} for existing projects; - Make your application ready to offer a RESTful interface, e.g. for consuming JSON content via AJAX or mobile clients - see @86@; - New tables may be defined via the ORM/ODM features of {\i mORMot}, still hosted in your external SQL server - see @27@, as any previous data; in particular, mixed pure-ORM and regular-SQL requests may coexist; or {\i mORMot}'s data modeling may balance your storage among several servers (and technologies, like NoSQL); - Sharing the same tables between legacy code SQL and {\i mORMot} ORM is possible, but to avoid consistency problems, you should better follow some rules detailed @106@; - You may benefit from our very fast in-memory engine, a dedicated {\i @*SQLite3@}-based consolidation database or even the caching features - see @38@, shared on the server side, when performance is needed - it may help integrating some @*CQRS@ pattern ({\i Command Query Responsibility Segregation}) into your application via a @*REST@ful interface, and delegate some queries from your main database; - If you are still using an old version of {\i Delphi}, and can't easily move up due to some third party components or existing code base, {\i mORMot} will offer all the needed features to start ORM, N-Tier and SOA, starting with a {\i Delphi} 6 edition. {\i mORMot} implements the needed techniques for introducing what Michael Feathers calls, in his book {\i Working Effectively With Legacy Code}, a @**seam@. A seam is an area where you can start to cleave off some legacy code and begin to introduce changes. Even mocking abilities of {\i mORMot} - see @62@ - will help you in this delicate task - see @http://www.infoq.com/articles/Utilizing-Logging Do not forget that {\i Synopse}, as a company, is able to offer dedicated audit and support for such a migration. The sooner, the better. \page :123 FAQ Before you start going any further, we propose here below a simple @**FAQ@ containing the most frequent questions we received on our forums. First of all, take a look at the {\i keyword index} available at the very beginning of this document. The underlined entries target the main article(s) about a given concept or technical term. Feel free to give your feedback at @https://synopse.info/forum asking new questions or improving answers! {\b Your SAD doc is too long to read through in a short period.}\line Too much documentation can kill the documentation! But you do not need to read the whole document: most of it is a detailed description of every unit, object, or class. But the first part is worth reading, otherwise you are very likely to miss some main concepts or patterns. It just takes 15-30 minutes! Also read @204@ to find out in which direction you may need to go for writing your server code. Consider the slides available at @https://drive.google.com/folderview?id=0B0r8u-FwvxWdeVJVZnBhSEpKYkE {\b Where should I start?}\line Take a look at the {\i Architecture principles} @40@, then download and install the sources @44@, then compile and run the {\f1\fs20 TestSQL3.dpr} program. Check about @*ORM@ @3@, @*SOA@ @63@ and @*MVC@ @108@, then test the various samples (from the {\f1\fs20 SQLite3\\Samples} folder), especially 01, 02, 04, 11, 12, 14, 17, 26, 28, 30 and the {\f1\fs20 MainDemo}. {\b So far, I can see your {\i mORMot} fits most of the requirement, but seems only for Database Client-Server apps.}\line First of all, the framework is a {\i set of bricks}, so you can use it e.g. to build interface based services, even with no database at all. We tried to make its main features modular and uncoupled. {\b I am not a great fan of ORM, sorry, I still like SQL and have some experience of that. Some times sophisticated SQL query is hard to change to ORM code.}\line ORM can make development much easier; but you can use e.g. interface-based services and "manual" SQL statements - in this case, you have at hand @27@ classes in {\i mORMot}, which allow very high performance and direct export to JSON. {\b I am tempted by using an ORM, but {\i mORMot} forces you to inherit from a root {\f1\fs20 @*TSQLRecord@} type, whereas I'd like to use any kind of object.}\line We will discuss this in details @168@. Adding attributes to an existing class is tempting, but will pollute your code at the end, mixing persistence and business logic: see {\i @*Persistence Ignorance@} and {\i @*Aggregates@} @124@. The framework proposes a second level of {\i Object} mapping, allowing to persist any kind of {\i @*PODO@} ({\i Plain Old Delphi Object}), by defining @*CQRS@ services - see @167@. {\b I would like to replace pieces of delphi-code by using mORMot and the @*DDD@-concept in a huge system, but its legacy database doesn't have integer primary keys, and {\i mORMot} ORM expects a {\f1\fs20 TID}-like field.}\line By design, such legacy tables are not compatible with {\i SQLite3} virtual tables, or our ORM - unless you add an {\f1\fs20 ID} integer additional primary key, which may not be the best idea. Some hints: write a {\i persistence service} as {\f1\fs20 interface}/{\f1\fs20 class} (as required by DDD - see @124@); uncouple persistence and @*SOA@ services (i.e. the SOA {\f1\fs20 @*TSQLRestServer@} is a {\f1\fs20 @*TSQLRestServerFullMemory@} and not a DB/ORM {\f1\fs20 @*TSQLRestServerDB@}); reuse your existing SQL statements, with {\f1\fs20 SynDB} as access layer if possible (you will have better performance, and direct @*JSON@ support); use the ORM for @*MicroService@ local persistence (with {\i SQLite3}), and/or for new tables in your legacy DB (or another storage, e.g. @*MongoDB@). {\b Why are you not using the latest features of the compiler, like generics or class attributes?}\line Our framework does not rely on {\i generics}, but on the power of the object pascal type system: specifying a {\f1\fs20 class} or {\f1\fs20 interface} type as parameter is safe and efficient - and generics tends to blow the executable size, lower down performance (the current RTL is not very optimized, and sometimes bugged), and hide implementation details. Some methods are available for newer version of the compiler, introducing access via generics; but it was not mandatory to depend on them. We also identified, as several Java or C# gurus, that {\f1\fs20 class} attributes may sound like a good idea, but tend to {\i pollute} the code, and introduce unexpected coupling. Last but not least, those features are incompatible with older version of Delphi we would like to support, and may reduce compatibility with @*FPC@. {\b I also notice in your SAD doc, data types are different from Delphi. You have {\f1\fs20 RawUTF8}, etc, which make me puzzled, what are they?}\line You can for sure use standard {\i Delphi} {\f1\fs20 string} types, but some more optimized types were defined: since the whole framework is @*UTF-8@ based, we defined a dedicated type, which works with all versions of {\i Delphi}, before and after {\i Delphi} 2009. By the way, just search for {\f1\fs20 RawUTF8} in the {\i keyword index} of this document, or see @32@. {\b During my tests, my client receives non standard @*JSON@ with unquoted fields.}\line Internally, the framework uses JSON in {\i MongoDB} @*extended syntax@, i.e. fields are not quoted - this gives better performance and reduces memory and bandwidth with a {\i mORMot} client. To receive {\f1\fs20 "field":value} instead of {\f1\fs20 field:value}, just add a proper {\f1\fs20 @**User-Agent@} HTTP header to the client request (as any browser does), and the server will emit standard JSON. {\b When I work with floating points and JSON, sometimes numerical values with more than 4 decimals are converted into JSON strings.}\line By default, {\f1\fs20 double} values are disabled in the JSON serialization, to avoid any hidden precision lost during conversion: see @194@ how to enable it. {\b I got an access violation with SynDB ISQLDBRows.}\line You need to explicitly release the {\f1\fs20 ISQLDBRows} instance, by setting it to {\f1\fs20 nil}, {\i before} freeing the owner's connection - see @195@. {\b @*Deadlock@ occurs with interface callbacks.}\line When working with asynchronous notifications over {\i WebSockets}, you need to ensure you won't fire directly a callback from a main method execution - see @196@ for several solutions. {\b All the objects seem non-VCL components, meaning need code each property and remember them all well.}\line This is indeed... a feature. The framework is not @*RAD@, but fully object-oriented. Thanks to the {\i Delphi} IDE, you can access all properties description via auto-completion and/or code navigation. We tried to make the documentation exhaustive and accurate. Then you can still use RAD for UI design, but let business be abstracted in pure code. See e.g. the {\f1\fs20 mORMotVCL.pas} unit which can publish any ORM result as {\f1\fs2 TDataSource} for your UI. {\b I know you have joined the {\i DataSnap} performance discussion and your performance won good reputation there. If I want to use your framework to replace my old project of DataSnap, how easy will it be?}\line If you used {\i DataSnap} to build method-based services, translation into {\i mORMot} will be just a matter of code refactoring. And you will benefit of new features like {\i Interface-based services} - see @63@ - which is much more advanced than the method-based pattern, and will avoid generating the client class via a wizard, and offers additional features - see @77@ or @72@. {\b What is the SMS? Do you know any advantage compared to JQuery or AngularJS?}\line {\i @*Smart Mobile Studio@} is an IDE and some source runtime able to develop and compile an Object-Pascal project into a {\i @*HTML 5@ / @*CSS 3@ / @*JavaScript@} {\i embedded} application, i.e. able to work stand alone with no remote server. When used with {\i mORMot} on the server side, you can use the very same object pascal language on both server and client sides, with strong typing and true @*OOP@ design. Then you feature secure authentication and JSON communication, with connected or off-line mode. Your {\i SmartPascal} client code can be generated by your {\i mORMot} server, as stated @90@. We currently focus on TMS Web Core integration, which seems a newer - and more supported - alternative. {\b I am trying to search a substitute solution to WebSnap. Do you have any sample or doc to describe how to build a robust web Server?}\line You can indeed easily create a modern @*MVC@ / @*MVVM@ scaling @*Web Application@. Your {\i mORMot} server can easily publish its ORM / SOA business logic as {\i Model}, use {\i @*Mustache@} logic-less templates rendering - see @81@ - for {\i Views}, and defining the {\i ViewModel} / {\i Controller} as regular Delphi methods. See @108@ for more details, and discovering a sample "blog" application. {\b Have you considered using a popular source coding host like @*Github@ or BitBucket?}\line We love to host our own source code repository, and find fossil a perfect match for our needs, with a friendly approach. But we created a parallel repository on {\i GitHub}, so that you may be able to monitor or fork our projects - see @https://github.com/synopse/mORMot \line Note that you can get a daily snapshot of our official source code repository directly from\line @https://synopse.info/files/mORMotNightlyBuild.zip {\b Why is this framework named {\i mORMot}?}\line - Because its initial identifier was "{\i Synopse SQLite3 database framework}", which may induce a {\i SQLite3}-only library, whereas the framework is now able to connect to any database engine;\line - Because we like mountains, and those large ground rodents;\line - Because marmots do hibernate, just like our precious objects;\line - Because marmots are highly social and use loud whistles to communicate with one another, just like our applications are designed not to be isolated;\line - Because even if they eat greens, they use to fight at Spring for their realm;\line - Because it may be an acronym for "Manage Object Relational Mapping Over Territory", or whatever you may think of... \page :40Architecture principles %cartoon08.png This framework tries to implement some "best-practice" patterns, among them: - {\i Model-View Controller} - see @10@; - {\i Multi-@*tier@ architecture} - see @7@; - {\i @*Test@-Driven Design} - see @12@; - {\i @*Stateless@} @*CRUD@/@*REST@ - see @9@; - {\i Object-Relational Mapping} - see @13@; - {\i Object-Document Mapping} - see @82@; - {\i @*Service@-Oriented Architecture} - see @17@. All those points render possible any project implementation, up to complex {\i @*Domain-Driven@} design - see @54@. \page : General design A general design of the {\i mORMot} architecture is shown in the following diagram: \graph mORMotDesign3 General mORMot architecture - Client Server implementation \ RESTful Client\RESTful Server\REST¤JSON \RESTful Client\RESTful Server\REST¤JSON subgraph cluster_1 { label=" mORMot Server"; \MVC/MVVM¤Web Server\RESTful Server \RESTful Server\Business¤rules\uses \Business¤rules\ORM \Services¤implementation\ORM \Authentication¤(users, sessions)\ORM \RESTful Server\Authentication¤(users, sessions)\requires \RESTful Server\external tables\makes¤CRUD \RESTful Server\in-memory tables¤JSON or binary files\makes¤CRUD \RESTful Server\NoSQL¤engine \RESTful Server\SQlite3¤engine\REST to SQL \RESTful Server\Services¤implementation\runs \SQlite3¤engine\in-memory tables¤JSON or binary files \SQlite3¤engine\external tables \SQlite3¤engine\SQLite3 data file\makes¤CRUD \ORM\SQlite3¤engine \ORM\in-memory tables¤JSON or binary files \ORM\external tables \ORM\NoSQL¤engine \ORM\Redirected¤TSQLRestServer¤ORM \ORM\Redirected¤TSQLRestClient } \external tables\External¤Database 1\SQL \external tables\External¤Database 2 \NoSQL¤engine\MongoDB subgraph cluster_2 { "External\nDatabase 1"; } subgraph cluster_3 { "External\nDatabase 2"; } subgraph cluster_4 { "MongoDB"; } subgraph cluster_5 { \Redirected¤TSQLRestClient\Remote¤mORMot¤Server\REST¤JSON } \Desktop Browser\MVC/MVVM¤Web Server\HTTP¤HTML \Mobile Browser\MVC/MVVM¤Web Server subgraph cluster_6 { label="Web Clients"; "Desktop Browser"; "Mobile Browser"; } subgraph cluster_7 { label="REST Client (FPC/FMX/Smart/Java/C#..)"; \ Services\ RESTful Client \ORM methods ¤over TSQLRecord\ RESTful Client\requests \User Interface \ORM methods ¤over TSQLRecord\asks \User Interface \ Services } subgraph cluster_8 { label="mORMot Client (Delphi)"; \Services\RESTful Client \RESTful Client\Business rules\uses \User Interface\Business rules\uses \Reporting\ORM methods¤over TSQLRecord\asks \ORM methods¤over TSQLRecord\RESTful Client\requests \User Interface\ORM methods¤over TSQLRecord\asks \User Interface\Services \User Interface\Reporting\runs \Services=ORM methods¤over TSQLRecord } \ ;subgraph cluster_8 { ;label="Client (Java/C#/JavaScript)"; ;\ Services\RESTful Client ;\REST Resources\RESTful Client\requests ;\User Interface\REST Resources\uses ;\User Interface\ Services ;} In addition, you may use the following transversal features: \graph mORMotDesign4 General mORMot architecture - Cross-Cutting features subgraph cluster_9 { label="Cross-Cutting features"; "File process\nCompression"; "Security\nCryptography"; "Remote access\nObjects cache"; "JSON\nUnicode\nUTF-8"; "Logging\nUnit Testing\nMocking"; } \ Don't be afraid. Such a drawing may sound huge and confusing, especially when you have a RAD background, and did not work much with modern design patterns. Following pages will detail and explain how the framework implements this architecture, and sample code is available to help you discovering the amazing {\i mORMot} realm. In the previous diagram, you can already identify some key concepts of {\i mORMot}: - Cross-Platform, multi clients, and multi devices; - Can integrate to an existing code base or architecture; - Client-Server RESTful design; - Layered (multi-tier) implementation; - Process can be defined via a set of Services (SOA); - Business rules and data model are shared by Clients and Server; - Data is mapped by objects (ORM/ODM); - Databases can be an embedded {\i SQLite3}, one or several standard RDBMS (with auto-generated SQL), a {\i MongoDB} NoSQL engine, fast in-memory objects lists, or another {\i mORMot} server; - Security (authentication and authorization) is integrated to all layers; - User interface and reporting classes are available; - You can write a MVC/MVVM AJAX or Web Application from your ORM/SOA methods; - Based on simple and proven patterns (REST, JSON, MVC, SOLID); - A consistent testing and debugging API is integrated; - Optimized for both scaling and stability. \page : Architecture Design Process First point is to state that you can't talk about architecture in isolation. Architecture is always driven by the actual needs of the application, not by whatever the architect read about last night and is dying to see how it works in the real world. There is no such "one architecture fits all" nor "one framework fits all" solution. Architecture is just a thinking of {\i how} you are building your own software. In fact, software architecture is not about theory and diagrams, nor just about best practice, but about a way of implementing a working solution for your customers. \graph ArchiIteration Architecture Iterative Process (SCRUM) subgraph cluster_0 { label="Customer"; \Software\Client } subgraph cluster_1 { label=" Team"; \Client\Use Cases\Requirements subgraph cluster_2 { label="BackLog"; "Use Cases"; } subgraph cluster_3 { label="Design" \Use Cases\Architecture¤Proposal \Use Cases\Risk¤Assessment \Use Cases\Technology¤& Models } subgraph cluster_4 { label=" Dev" \Risk¤Assessment\Tasks \Architecture¤Proposal\Tasks \Technology¤& Models\Tasks } } \Tasks\Software\Definition¤of Done \ This diagram presents how Architecture is part of a typical SCRUM iterative agile process. Even if some people of your company may be in charge of global software architecture, or even if your project managements follows a classic V-cycle and does not follow the agile manifesto, architecture should never be seen as a set of rules, to be applied by every and each developers. Architecture is part of the coding, but not all the coding. Here are some ways of achieving weak design: - Let each developer decides, from his/her own knowledge (and mood?), how to implement the use cases, with no review, implementation documentation, nor peer collaboration; - Let each team decides, from its own knowledge (and untold internal leadership?), how to implement the use cases, with no system-wide collaboration; - Let architecture be decided at so high level that it won't affect the actual coding style of the developers (just don't be caught); - Let architecture be so much detailed that each code line has to follow a typical implementation pattern, therefore producing over engineered code; - Let architecture map the existing, with some middle-term objectives at best; - Let technology, frameworks or just-blogged ideas be used with no discrimination (do not trust the sirens of dev marketing). Therefore, some advices: - Collaboration is a need - no one is alone, no team is better, no manager is always right; - Sharing is a need - between individuals, as teams, with managers; - Stay customer and content focused; - Long term is prepared by today's implementation; - Be lazy, i.e. try to make tomorrow's work easier for you and your team-workers; - They did not know it was impossible, so they did it. Purpose of frameworks like {\i mORMot} is to provide your teams with working and integrated set of classes, so that you can focus on your product, enjoying the collaboration with other Open Source users, in order to use evolving and pertinent software architecture. \page :10 Model-View-Controller The {\i Model-View-Controller} (@**MVC@) is a software architecture, currently considered an architectural pattern used in software engineering. The pattern isolates "domain logic" (the application logic for the user) from the user interface (input and presentation), permitting independent development, testing and maintenance of each (separation of concerns). \graph MVCModel1 Model View Controller process rankdir=LR; \Controller\Model\Use \Controller\View\Refresh \Model\Controller\Notify updates \View\Controller\Command \ The {\b @*Model@} manages the behavior and data of the application domain, responds to requests for information about its state (usually from the view), and responds to instructions to change state (usually from the controller). In @*Event-Driven@ systems, the model notifies observers (usually views) when the information changes so that they can react - but since our ORM is @*stateless@, it does not need to handle those events - see @15@. The {\b View} renders the model into a form suitable for interaction, typically a user interface element. Multiple views can exist for a single model for different purposes. A {\i viewport} typically has a one to one correspondence with a display surface and knows how to render to it. The {\b Controller} receives user input and initiates a response by making calls on model objects. A controller accepts input from the user and instructs the model and viewport to perform actions based on that input. \graph MVCModel2 Model View Controller concept rankdir=LR; \Model\View\indirect¤association \Controller\Model\direct¤association \View\Controller\indirect¤association \Controller\View\direct¤association \ In the framework, the {\i model} is not necessarily merely a database; the {\i model} in MVC is both the data and the business/domain logic needed to manipulate the data in the application. In our ORM, a model is implemented via a {\f1\fs20 @*TSQLModel@} class, which centralizes all {\f1\fs20 @*TSQLRecord@} inherited classes used by an application, both database-related and business-logic related. The {\i views} can be implemented using: - For Desktop clients, a full set of User-Interface units of the framework, which is mostly auto-generated from code - they will consume the {\i model} as reference for rendering the data; - For Web clients, an integrated high-speed {\i @*Mustache@} rendering engine - see @81@ - is able to render HTML pages with logic-less templates, and {\i controller} methods written in Delphi - see @108@; - For @*AJAX@ clients, the server side is easy to be reached from @*REST@ful @*JSON@ services. The {\i controller} is mainly already implemented in our framework, within the RESTful commands, and will interact with both the associated {\i view} (e.g. for refreshing the User Interface) and {\i model} (for data handling). Some custom actions, related to the business logic, can be implemented via some custom {\f1\fs20 TSQLRecord} classes or via custom RESTful @*Service@s - see @11@. \page :7 Multi-tier architecture In software engineering, multi-@**tier@ architecture (often referred to as {\i n-tier} architecture) is a client–server architecture in which the presentation, the application processing, and the data management are logically separate processes. For example, an application that uses middle-ware to service data requests between a user and a database employs multi-tier architecture. The most widespread use of multi-tier architecture is the three-tier architecture. In practice, a typical VCL/FMX RAD application written in Delphi has a two-tier architecture: \graph ArchiTwoTier Two-Tier Architecture - Logical View \Application Tier\Data Tier \ In this approach, the {\i Application Tier} mixes the UI and the logic in forms and modules. Both @*ORM@ and @*SOA@ aspects of our @*REST@ful framework make it easy to develop using a more versatile three-tier architecture. \graph ArchiMultiTier Multi-Tier Architecture - Logical View \Presentation Tier\Logic Tier \Logic Tier\Data Tier \ The {\i Synopse mORMot Framework} follows this development pattern: - {\i Data Tier} is either {\i @*SQLite3@} and/or an internal very fast in-memory database; most @*SQL@ queries are created on the fly, and database table layout are defined from {\i Delphi} classes; you can also use any external database, currently {\i SQLite3, @*Oracle@, @*Jet/MSAccess@, @*MS SQL@, @*Firebird@, @*DB2@, @*PostgreSQL@, @*MySQL@, @*Informix@} and {\i @*NexusDB@} SQL dialects are handled, and even @*NoSQL@ engines like {\i @*MongoDB@} can be directly used - see @27@; - {\i Logic Tier} is performed by pure ORM aspect and SOA implementation: you write {\i Delphi} classes which are mapped by the {\i Data Tier} into the database, and you can write your business logic as Services called as {\i Delphi} {\f1\fs20 interface}, up to a {\i @*Domain-Driven@} design - see @54@ - if your project reaches some level of complexity; - {\i Presentation Tier} is either a {\i Delphi} Client, or an @*AJAX@ application, because the framework can communicate using @*REST@ful @*JSON@ over @*HTTP@/1.1 (the {\i Delphi} Client User Interface is generated from Code, by using @*RTTI@ and structures, not as a RAD - and the Ajax applications need to be written by using your own tools and @*JavaScript@ framework, there is no "official" Ajax framework included yet). In fact, {\i mORMot} can scales up to a {\i @*Domain-Driven@} Design four-tier architecture - see @54@ - as such: - {\i Presentation Tier} which can be e.g. a {\i Delphi} or AJAX client; - {\i Application Tier} which serves JSON content according to the client application; - {\i Business Logic Tier} which centralizes all the {\i Domain} processing, shared among all applications; - {\i Persistence/Data Tier} which can be either in-process (like {\i @*SQLite3@} or in-memory) or external (e.g. {\i Oracle, MS SQL, DB2, PostgreSQL, MySQL, Informix}...). \graph ArchiDDDTier Domain Driven Design n-Tier Architecture - Logical View \Presentation Tier\Application Tier \Application Tier\Business Logic Tier \Business Logic Tier\Data Tier \ Note that you have to make a difference between {\i physical} and {\i logical} n-tier architecture. Most of the time, n-Tier is intended to be a {\i physical} (hardware) view, for instance a separation between the database server and the application server, placing the database on a separate machine to facilitate ease of maintenance. In {\i mORMot}, and more generally in SOA - see @17@, we deal with {\i logical} layout, with separation of layers through interfaces - see @46@ - and the underlying hardware implementation will usually not match the logical layout. \graph ArchiDDDPhysical Domain Driven Design n-Tier Architecture - Physical View subgraph cluster_0 { label="Client 1 (Delphi)"; "Presentation Tier"; } subgraph cluster_3 { label="Client 2 (AJAX)"; "Presentation Tier "; } subgraph cluster_1 { label="Application Server"; \Application Tier\Business Logic Tier } \Presentation Tier \Application Tier \Presentation Tier\Application Tier subgraph cluster_2 { label=" DB Server"; \Business Logic Tier\Data Tier } \ In this document, we will focus on the logical way of thinking / coding, letting the physical deployment be made according to end-user expectations. \page :17 Service-Oriented Architecture (SOA) @*Service@-Oriented Architecture (@**SOA@) is a flexible set of design principles used during the phases of systems development and integration in computing. A system based on a SOA will package functionality as a suite of inter-operable services that can be used within multiple, separate systems from several business domains. A software service is a logicical representation of a repeatable activity that produce a precise result. In short, a {\i consumer} ask to a {\i producer} to act in order to produce a {\i result}. Most of the time, this invocation is free from any previous invocation (it is therefore called {\i stateless}). The SOA implementations rely on a mesh of software services. Services comprise unassociated, {\i loosely coupled} units of functionality that have no calls to each other embedded in them. Each service implements {\i one action}, such as filling out an online application for an account, or viewing an online bank statement, or placing an online booking or airline ticket order. Rather than services embedding calls to each other in their source code, they use defined protocols that describe how services pass and parse messages using description meta-data. \graph ArchiSOA Service Oriented Architecture - Logical View rankdir=LR; subgraph cluster0_ { label="Consumers"; "Client A"; "Client B"; "Client C"; } subgraph cluster1_ { label="Service Bus"; \Client A\Publisher 1 \Client A\Publisher 2 \Client B\Publisher 2 \Client B\Publisher 3 \Client C\Publisher 3 } subgraph cluster2_ { label="Publishers"; \Publisher 1\Service 1 \Publisher 2\Service 2 \Publisher 3\Service 3 \Service 3\Publisher 2 } \ Since most of those services are by definition @*stateless@, some kind of {\i service composition} is commonly defined to provide some kind of logical multi-tier orchestration of services. A higher level service invokes several services to work as a self-contained, stateless service; as a result, lower-level services can still be stateless, but the consumer of the higher level service is able to safely process some kind of transactional process. \graph ArchiSOAComposition Service Oriented Architecture - Logical View of Composition rankdir=LR; subgraph cluster0_ { label="Consumers"; "Client A"; } subgraph cluster1_ { label="Application Service Bus"; \Client A\Composition¤Publisher } subgraph cluster2_ { label="Application Publishers"; \Composition¤Publisher\Composition¤Service } subgraph cluster3_ { label="Business Service Bus"; \Composition¤Service\Publisher 1 \Composition¤Service\Publisher 2 \Composition¤Service\Publisher 3 } subgraph cluster4_ { label="Business Publishers"; \Publisher 1\Service 1 \Publisher 2\Service 2 \Publisher 3\Service 3 \Service 3\Publisher 2 } \ For more details about SOA, see @http://en.wikipedia.org/wiki/Service-oriented_architecture SOA is mainly about {\i decoupling}.\line That is, it enables implementation independence in a variety of ways, for instance: |%15%40%40 |\b Dependency|Desired decoupling|Decoupling technique\b0 |Platform|Hardware, Framework or Operating System should not constrain choices of the Services consumers|Standard protocols, mainly Web services (e.g. SOAP or RESTful/JSON) |Location|Consumers may not be impacted by service hosting changes|Routing and proxies will maintain Services access |Availability|Maintenance tasks shall be transparent|Remote access allows centralized support on Server side |Versions|New services shall be introduced without requiring upgrades of clients|Contract marshalling can be implemented on the Server side |% SOA and ORM - see @13@ - do not exclude themselves. In fact, even if some software architects tend to use only one of the two features, both can coexist and furthermore complete each other, in any @*Client-Server@ application: - ORM access could be used to access to the data with objects, that is with the native presentation of the Server or Client side ({\i Delphi}, {\i @*JavaScript@}...) - so ORM can be used to provide efficient access to the data or the business logic - this is the idea of @*CQRS@ pattern; - SOA will provide a more advanced way of handling the business logic: with custom parameters and data types, it is possible to provide some high-level Services to the clients, hiding most of the business logic, and reducing the needed bandwidth. In particular, SOA will help leaving the business logic on the Server side, therefore will help increasing the @7@. By reducing the back-and-forth between the Client and the Server, it will also reduce the network bandwidth, and the Server resources (it will always cost less to run the service on the Server than run the service on the Client, adding all remote connection and @*serialization@ to the needed database access). Our @*interface@-based SOA model allows the same code to run on both the client and the server side, with a much better performance on the server side, but a full interoperability of both sides. \page :13 Object-Relational Mapping (ORM) In practice, @**ORM@ gives a set of methods to ease high-level objects persistence into a @*RDBMS@. Our {\i Delphi} {\f1\fs20 class} instances are not directly usable with a relational database, which is since decades the most convenient way of persisting data. So some kind of "glue" is needed to let class properties be saved into one or several tables. You can interact with the database using its native language, aka SQL. But SQL by itself is a full programming language, with diverse flavors depending on the exact backend engine (just think about how you define a column type able to store text). So writing and maintaining your SQL statements may become a time-consuming, difficult and error-prone task. Sometimes, there will be nothing better than a tuned SQL statement, able to aggregate and join information from several tables. But most of the time, you will need just to perform some basic operations, known as @**CRUD@ (for {\i Create Retrieve Update Delete} actions) on well identified objects: this is where ORM may give you a huge hint, since it is able to generate the SQL statements for you. The ORM works in fact as such: \graph mORMotORMprocess ORM Process \object¤instance\ORM\CRUD¤operations \ORM\SQL\mapping \SQL\RDBMS\DB client \ORM\object¤instance \SQL\ORM \RDBMS\SQL \ The ORM core retrieve information to perform the mapping: - Object definition via its {\f1\fs20 class} type (via @*RTTI@); - Database model as retrieved for each database engine. \graph mORMotORMmapping ORM mapping \ORM\class type \ORM\data model \class type\object instance \data model\RDBMS \ Since several implementation schemes are possible, we will first discuss the pros and the cons of each one. First, here is a diagram presenting some common implementation schemes of database access with {\i Delphi} (which maps most other languages or frameworks, including C# or Java). \graph mORMotORM Why a Client-Server ORM \UI Components\DataBase\RAD \UI\Delphi classes\code mapping \Delphi classes\DataBase \Handwritten¤SQL \UI \ORM\MVC¤binding \ORM\ Database \Generated¤SQL \ UI 1\Client 1\MVC/MVVM¤binding \ UI 2 (web)\Client 2\MVC¤binding \Client 1\Server\Secure¤protocol¤(REST) \Client 2\Server \Server\ORM \Persistence¤layer \ORM \ Database \Generated¤SQL \ \page The table below is a very suggestive (but it doesn't mean wrong) {\i Resumé} of some common schemes, in the {\i Delphi} world. @*ORM@ is just one nice possibility among others. |%17%40%44 |\b\qc Scheme|Pros|Cons\b0 |Use DB views and tables, with GUI components|- @*SQL@ is a powerful language\line - Can use high-level DB tools (UML) and RAD approach|- Business logic can't be elaborated without stored procedures\line - SQL code and stored procedures will bind you to a DB engine\line - Poor Client interaction\line - Reporting must call the DB directly\line - No Multi-tier architecture |Map DB tables or views with {\i Delphi} classes|- Can use elaborated business logic, in {\i Delphi}\line - Separation from UI and data|- SQL code must be coded by hand and synchronized with the classes\line - Code tends to be duplicated\line - SQL code could bind you to a DB engine\line - Reports can be made from code or via DB related tools\line - Difficult to implement true Multi-tier architecture |Use a Database ORM|- Can use very elaborated business logic, in {\i Delphi}\line - SQL code is generated (in most cases) by the ORM\line - ORM will adapt the generated SQL to the DB engine|- More abstraction needed at design time (no RAD approach)\line - In some cases, could lead to retrieve more data from DB than needed\line - Not yet a true Multi-tier architecture, because ORM is for DB access only and business logic will need to create separated classes |Use a @*Client-Server@ ORM|- Can use very elaborated business logic, in {\i Delphi}\line - SQL code is generated (in most cases) by the ORM\line - ORM will adapt the generated SQL to the DB engine\line - Services will allow to retrieve or process only needed data\line - Server can create objects viewed by the Client as if they were DB objects, even if they are only available in memory or the result of some business logic defined in {\i Delphi}\line - Complete Multi-tier architecture|- More abstraction needed at design time (no RAD approach) |% Of course, you'll find out that our framework implements a Client-Server ORM, which can be down-sized to @*stand-alone@ mode if needed, but which is, thanks to its unique implementation, scalable to any complex Domain-Driven Design. As far as we found out, looking at every language and technology around, almost no other ORM supports such a native Client-Server orientation. Usual practice is to use a @17@ for remote access to the ORM. Some projects allow remote access to an existing ORM, but they are separated projects. Our {\i mORMot} is pretty unique, in respect to its @*REST@ful Client-Server orientation, from the ground up. If you entered the {\i Delphi} world years ago, you may be pretty fluent with the RAD approach. But you probably also discovered how difficult it is to maintain an application which mixes UI components, business logic and database queries. Today's software users have some huge ergonomic expectations about software usability: some screens with grids and buttons, mapping the database, won't definitively be appealing. Using {\i mORMot}'s ORM /SOA approach will help you focus on your business and your clients expectations, letting the framework perform most of the plumbing for you. \page :82 NoSQL and Object-Document Mapping (ODM) @**SQL@ is the De-Facto standard for data manipulation - Schema-based; - Relational-based; - ACID by transactions; - Time proven and efficient; - "Almost" standard (each DB has its own column typing system). {\i @**NoSQL@} is a new paradigm, named as such in early 2009 (even if some database engines, like {\i Lotus Domino}, may fit the definition since decades): - {\i NoSQL} stands for "Not Only SQL" - which is more positive than "no SQL"; - Designed to scale for the web and @*BigData@ (e.g. Amazon, Google, Facebook), e.g. via easy @*replication@ and simple API; - Relying on no standard (for both data modeling and querying); - A lot of diverse implementations, covering any data use - @http://nosql-database.org lists more than 150 engines. We can identify two main families of NoSQL databases: - {\i Graph}-oriented databases; - {\i Aggregate}-oriented databases. {\i Graph}-oriented databases store data by their relations / associations: \graph NoSQLGraphDB NoSQL Graph Database \Alice\Bob\ID: 100¤Label: knows¤Since: 2001/10/03 \Bob\Alice\ID: 101¤Label: knows¤Since: 2001/10/04 \Alice\Group\ID: 102¤Label: is_member¤Since: 2005/07/01 \Group\Alice\ID: 103¤Label: members \Bob\Group\ID: 105¤Label: is_member¤Since: 2011/02/14 \Group\Bob\ID: 104¤Label: members =Alice=ID: 1¤Name: Alice¤Age: 18 =Bob=ID: 2¤Name: Bob¤Age: 22 =Group=ID: 3¤Type: Group¤Name: Chess \ Such kind of databases are very useful e.g. for developing any "social" software, which will value its data by the relations between every node. Such data model does not fit well with the relational model, whereas a {\i NoSQL} engine like {\i Neo4j} handles such kind of data natively. Note that by design, {\i Graph}-oriented databases are ACID. But the main {\i NoSQL} database family is populated by the {\i Aggregate}-oriented databases. By {\i Aggregate}, we mean the same definition as will be used @54@ for {\i Domain Driven Design}. It is a collection of data that we interact with as a unit, which forms the boundaries for ACID operations in a given model. In fact, {\i Aggregate}-oriented databases can be specified as three main implementation/query patterns: - Document-based (e.g. {\i @*MongoDB@, CouchDB, RavenDB}); - Key/Value (e.g. {\i Redis, Riak, Voldemort}); - Column family (e.g. {\i Cassandra, HiBase}). Some of them can be {\i schema-less} (meaning that the data layout is not fixed, and can evolve on the fly without re-indexing the whole database) - but column-driven bases do have a schema, or even storing plain BLOB of data (this is the purpose of Key/Value engines, which focus on storage speed and rely on the client side to process the data). In short, RDBMS stores data per table, and need to JOIN the references to get the aggregated information: \graph NoSQLAggregJoin SQL Aggregate via JOINed tables rankdir=LR; node [shape=Mrecord]; struct1 [label="ID|UserName"]; struct2 [label="ID|UserID|Phone|Email"]; struct3 [label="ID|UserID|Level|Group"]; struct2:f0 -> struct1; struct3:f0 -> struct1; \ Whereas {\i NoSQL} stores its aggregates as documents: the whole data is embedded in one. \graph NoSQLAggregNoSQL NoSQL Aggregate as one document rankdir=LR; node [shape=Mrecord]; struct1 [label="ID|UserName|Contact.Phone|Contact.Email|Access.Level|Access.Group"]; \ Which may be represented as the following @*JSON@ - see @2@ - data: µ{ µ "ID": 1234, µ "UserName": "John Smith", µ "Contact": { µ "Phone": "123-456-789", µ "Email": "xyz@abc.com" µ }, µ "Access": { µ "Level": 5, µ "Group": "dev" µ } µ} Such a document will fit directly with the object programming model, without the need of thinking about JOINed queries and database plumbing. As a result, we can discuss the two data models: - {\i Relational data Model} with highly-structured table organization, and rigidly-defined data formats and record structure; - {\i Document data Model} as a collection of complex documents with arbitrary, nested data formats and varying "record" format. The {\i Relational} model features {\i @**normalization@} of data, i.e. organize the fields and tables of a relational database to minimize redundancy.\line On the other hand, the {\i Document} model features {\i @**denormalization@} of data, to optimize the read performance of a database by adding redundant data or by grouping data. It also features horizontal scaling of the servers, since data can easily be balanced among several servers, without the speed penalty of performing a remote JOIN. One of the main difficulties, when working with {\i NoSQL}, is to define how to {\i denormalize} the data, and when to store the data in {\i normalized} format.\line One good habit is to model your data depending on the most current queries you will have to perform. For instance, you may embed sub-documents which will be very likely to be requested by your application most of the time. Note that most {\i NoSQL} engines feature a {\i projection} mechanism, which allows you to return only the needed fields for a query, leaving the sub-documents on the server if you do not need them at this time. The less frequent queries may be executed over separated collections, populated e.g. with consolidated information.\line Since {\i NoSQL} databases have fewer hard-and-fast rules than their relational databases ancestors, you are more likely to tune your model, depending on your expectations. In practice, you may spend less time thinking about "how" to store the data than with a RDBMS, and are still able to {\i normalize} information later, if needed. {\i NoSQL} engines do not fear redundant information, as soon as you follow the rules of letting the client application take care of the whole data consistency (e.g. via one ORM). As you may have stated, this {\i Document data Model} is much closer to the @*OOP@ paradigm than the classic relational scheme. Even a new family of frameworks did appear together with {\i NoSQL} adoption, named {\i Object Document Mapping} (@**ODM@), which is what @13@ was for RDBMS. In short, both approaches have benefits, which are to be weighted. |%50%50 |\b SQL|NoSQL\b0 |Ubiquitous SQL|Map OOP and complex types\line (e.g. arrays or nested documents) |Easy vertical scaling|Uncoupled data: horizontal scaling |Data size (avoid duplicates and with no schema)|Schema-less: cleaner evolution |Data is stored once, therefore consistent|Version management (e.g. {\i CouchDB}) |Complex ACID statements|Graph storage (e.g. {\i Redis}) |@*Aggregation@ functions (depends)|@*Map/Reduce@ or Aggregation functions\line (e.g. since {\i MongoDB} 2.2) |% With {\i mORMot}, you can switch from a classic SQL engine into a trendy {\i MongoDB} server, just in one line of code, when initializing the data on the server side. You can switch from ORM to ODM at any time, even at runtime, e.g. for a demanding customer. :54 Domain-Driven Design : Definition @http://domaindrivendesign.org gives the somewhat "official" definition of {\i @**Domain-Driven@} design (DDD): {\i Over the last decade or two, a philosophy has developed as an undercurrent in the object community. The premise of domain-driven design is two-fold:} - {\i For most software projects, the primary focus should be on the domain and domain logic;} - {\i Complex domain designs should be based on a model.} {\i Domain-driven design is not a technology or a methodology. It is a way of thinking and a set of priorities, aimed at accelerating software projects that have to deal with complicated domains.} Of course, this particular architecture is customizable according to the needs of each project. We simply propose following an architecture that serves as a baseline to be modified or adapted by architects according to their needs and requirements. : Patterns In respect to other kinds of @7@, DDD introduces some restrictive patterns, for a cleaner design: - Focus on the {\i @*Domain@} - i.e. a particular kind of knowledge; - Define {\i @*Bounded context@s} within this domain; - Create an evolving {\i @*Model@} of the domain, ready-to-be consumed by applications; - Identify some kind of objects - called {\i @*Value objects@} or {\i @*Entity Objects@} / {\i @*Aggregates@}; - Use an {\i @*Ubiquitous Language@} in resulting model and code; - Isolate the domain from other kind of concern (e.g. persistence should not be called from the domain layer - i.e. the domain should not be polluted by technical considerations, but rely on the {\i @*Factory@} and {\i @*Repository@} patterns); - Publish the domain as well-defined uncoupled {\i @*Service@s}; - Integrate the domain services with existing applications or legacy code. The following diagram is a map of the patterns presented and the relationships between them.\line It is inspired from the one included in the Eric Evans's reference book, {\i "@*Domain-Driven@ Design", Addison-Wesley}, 2004 (and updated to take in account some points appeared since). \graph ArchiDDDSyntax Domain-Driven Design - Building Blocks \MDD\Ubiquitous¤Language\define¤model with \MDD\Bounded¤Contexts\identify¤scope with \MDD\Services\process¤model with \MDD\Entities\express¤model with \MDD\VO\express¤model with \MDD\Clean/Layered/¤Hexagonal¤Architecture\isolate¤domain with \MDD\Events\express state¤changes with \MDD\RAD\exclude \Entities\Repositories\access with \Entities\Aggregates\encapsulate with \VO\Factories\instantiated by \Entities\Factories\instantiated by \Aggregates\Factories\instantiated by \Aggregates\Repositories\access with \VO\Aggregates\encapsulate with =MDD=Model-Driven¤Design =VO=Value Objects \ You may recognize a lot of existing patterns you already met or implemented. What makes DDD unique is that those patterns have been organized around some clear concepts, thanks to decades of business software experiment. : Is DDD good for you? {\i Domain-Driven design} is not to be used everywhere, and in every situation. First of all, the following are prerequisite of using DDD: - Identified and well-bounded domain (e.g. your business target should be clearly identified); - You must have access to domain experts to establish a creative collaboration, in an iterative (may be {\i agile}) way; - Skilled team, able to write clean code - note also that since DDD is more about code expressiveness than technology, it may not appear so "trendy" to youngest developers; - You want your internal team to accumulate knowledge of the domain - therefore, outsourcing may be constrained to applications, not the core domain. Then check that DDD is worth it, i.e. if: - It helps you solving the problem area you are trying to address; - It meets your strategic goals: DDD is to be used where you will get your business money, and make you distinctive from your competitors; - You need to bring clarity, and need to solve inner complexity, e.g. modeling a lot of rules (you won't use DDD to build simple applications - in this case, RAD may be enough); - Your business is exploring: your goal is identified, but you do not know how to accomplish it; - Don't have all of these concerns, but at least one or two. : Introducing DDD Perhaps DDD sounds more appealing to you now. In this case, our {\i mORMot} framework will provide all the bricks you need to implement it, focusing on your domain and letting the libraries do all the needed plumbing.\line If you identified that DDD is not to be used now, you will always find with {\i mORMot} the tools you need, ready to switch to DDD when it will be necessary. @66@ will benefit from DDD patterns. Finding so-called @*seam@s, along with isolating your core domain, can be extremely valuable when using DDD techniques to refactor and tighten the highest value parts of your code. It is not mandatory to re-write your whole existing software with DDD patterns everywhere: once you have identified where your business strategy's core is, you can introduce DDD progressively in this area. Then, following continuous feedback, you will refine your code, adding regression tests, and isolating your domain code from end-user code. For a technical introduction about DDD and how {\i mORMot} can help you implement this design, see @68@. With {\i mORMot}, your software solution will never be stuck in a dead-end. You'll be able to always adapt to your customers need, and maximize your ROI. ;:Global Architecture layout ;In the following sections, the {\i Synopse mORMot Framework} library architecture is detailed as: ;\SourcePage [SAD-Main] SourcePath=Lib\SQLite3 IncludePath=Lib;Lib\SQLite3;Lib\SynDBDataset;Lib\CrossPlatform;Lib\SQLite3\DDD\dom;Lib\SQLite3\DDD\infra;Zeos\src ;Lib\SQLite3\Samples\MainDemo SourceFile=TestSQL3.dpr;SynLog.pas;SynTests.pas;mORMot.pas;mORMoti18n.pas;mORMotToolBar.pas;mORMotUI.pas;mORMotUIEdit.pas;mORMotMVC.pas;mORMotUILogin.pas;mORMotReport.pas;mORMotUIOptions.pas;mORMotUIQuery.pas;mORMotService.pas;mORMotSQLite3.pas;mORMotHttpClient.pas;mORMotHttpServer.pas;SynSQLite3.pas;SynSQLite3Static.pas;SynSQLite3RegEx.pas;SynCrtSock.pas;SynCurl.pas;SynDB.pas;SynOleDB.pas;SynDBOracle.pas;SynDBSQLite3.pas;SynDBODBC.pas;SynDBDataset.pas;SynDBZeos.pas;SynDBFireDAC.pas;SynDBUniDAC.pas;SynDBBDE.pas;SynDBNexusDB.pas;SynDBVCL.pas;mORMotReport.pas;mORMotVCL.pas;mORMotDB.pas;mORMotFastCGIServer.pas;SynSM.pas;SynDBMidasVCL.pas;mORMotMidasVCL.pas;SynMongoDB.pas;SynFastWideString.pas;SynCrossPlatformJSON.pas;SynCrossPlatformREST.pas;SynCrossPlatformSpecific.pas;SynCrossPlatformTests.pas;dddInfraSettings.pas ;Samples\MainDemo\SynFile.dpr SourceIgnoreSymbol=select,check,open,connect,send,sqlite3,mORMot,JavaScript,cypher,execute,cache SourceIgnoreSymbolByUnit=SynCrossPlatformJSON,SynCrossPlatformREST,SynCrossPlatformSpecific,SynCrossPlatformTests Version=1.18 TitleOffset=0 DisplayName=mORMot Framework :Enter new territory : Meet the mORMot The {\i Synopse mORMot} framework consists in a huge number of units, so we will start by introducing them. \graph mORMotSourceCodeMainUnits mORMot Source Code Main Units node [shape="box"]; rankdir=LR; \mORMot.pas\SynCommons.pas \mORMot.pas\SynTests.pas \mORMot.pas\SynLog.pas \mORMotDB.pas\mORMot.pas \mORMotDB.pas\SynCommons.pas \mORMotDB.pas\SynDB.pas \UI\mORMot.pas \UI\SynCommons.pas \mORMotSQlite3.pas\SynCommons.pas \mORMotSQlite3.pas\mORMot.pas \mORMotSQlite3.pas\SynSQLite3.pas¤SynSQLite3Static.pas \SynSQLite3.pas¤SynSQLite3Static.pas\SynCommons.pas \SynDB.pas\SynCommons.pas \SynDB.pas\SynLog.pas \SynDBSQlite3.pas\SynDB.pas \SynDBSQlite3.pas\SynCommons.pas \SynDBSQlite3.pas\SynSQLite3.pas¤SynSQLite3Static.pas \SynDBOracle.pas¤SynDBODBC.pas¤SynOleDB.pas¤SynDBZEOS.pas¤SynDBDataset.pas¤SynDBFireDAC.pas¤SynDBUniDAC.pas¤SynDBNexusDB.pas¤SynDBBDE.pas¤SynDBRemote.pas\SynDB.pas \mORMotDB.pas\SynSQLite3.pas¤SynSQLite3Static.pas \mORMotReport.pas\SynCommons.pas \mORMotReport.pas\SynPdf.pas \SynPdf.pas\SynCommons.pas \SynSMAPI.pas¤SynSM.pas\SynCommons.pas \HTTP\SynCrtSock.pas \HTTP\mORMot.pas \HTTP\SynCommons.pas \mORMotMongoDB.pas\SynMongoDB.pas \mORMotMongoDB.pas\mORMot.pas \mORMotMVC.pas\mORMot.pas \mORMotMVC.pas\SynCommons.pas \mORMotMVC.pas\SynMustache.pas \SynMongoDB.pas\SynCommons.pas \SynMongoDB.pas\SynLog.pas \SynMongoDB.pas\SynCrtSock.pas =UI=mORMotUI.pas¤mORMotToolBar.pas¤mORMoti18n.pas¤mORMotUILogin.pas¤mORMotUIEdit.pas¤mORMotUIQuery.pas¤mORMotUIOptions.pas =HTTP=mORMotHttpClient.pas¤mORMotHttpServer.pas \mORMotReport.pas=UI=SynSMAPI.pas¤SynSM.pas \ ;When you will finish reading this document, you won't be afraid any more! :) \page : Main units The main units you have to be familiar with are the following: |%30%70 |\b Unit name|Description\b0 |{\f1\fs20 SynCommons.pas}\line {\f1\fs20 SynLog.pas}\line {\f1\fs20 SynTests.pas}|Common types, classes and functions |{\f1\fs20 mORMot.pas}|Main unit of the @*ORM@ / @*SOA@ framework |{\f1\fs20 SynSQLite3.pas\line SynSQLite3Static.pas}|{\i @*SQLite3@} database engine |{\f1\fs20 mORMotSQLite3.pas}|Bridge between {\f1\fs20 mORMot.pas} and {\f1\fs20 SynSQLite3.pas} |{\f1\fs20 @*SynDB@.pas\line SynDB*.pas}|Direct RDBMS access classes |{\f1\fs20 mORMotDB.pas}|ORM external {\f1\fs20 SynDB.pas} access, via {\i SQlite3} virtual tables |{\f1\fs20 SynMongoDB.pas\line mORMotMongoDB.pas}|Direct access to a {\i @*MongoDB@} server |{\f1\fs20 SynSM.pas}\line {\f1\fs20 SynSMAPI.pas}|{\i SpiderMonkey} JavaScript engine |{\f1\fs20 mORMotHttpClient.pas\line mORMotHttpServer.pas\line SynCrtSock.pas}|@*REST@ful HTTP/1.1 Client and Server |{\f1\fs20 mORMotMVC.pas\line SynMustache.pas}|@*MVC@ classes for writing @*Web Application@s |{\f1\fs20 mORMotUI*.pas}|Grid and Forms User Interface generation |{\f1\fs20 mORMotToolBar.pas}|ORM ToolBar User Interface generation |{\f1\fs20 mORMotReport.pas}|Integrated Reporting engine |% Other units are available in the framework source code repository, but are either expected by those files above (e.g. like {\f1\fs20 SynDB*.pas} database providers), or used only optionally in end-user @*cross-platform@ client applications (e.g. the {\f1\fs20 CrossPlatform} folder). In the following pages, the features offered by those units will be presented.\line Do not forget to take a look at all sample projects available in the {\f1\fs20 SQLite3\\Samples} sub-folders - nothing is better than some simple code to look at. Then detailed information will be available in the second part of this document - see @44@. :45SynCommons unit %cartoon01.png First of all, let us introduce some cross-cutting features, used everywhere in the {\i Synopse} source code. Even if you do not need to go deeply into the implementation details, it will help you not be disturbed with some classes and types you may encounter in the framework source, and its documentation. It was a design choice to use some custom low-level types, classes and functions instead of calling the official {\i Delphi} RTL.\line Benefits could be: - Cross-platform and cross-compiler support (e.g. leverage specificities, about memory model or RTTI); - Unicode support for all versions of {\i Delphi}, even before {\i Delphi} 2009, or with @*FPC@; - Optimized for process speed, multi-thread friendliness and re-usability; - Sharing of most common features (e.g. for text/data processing); - @*KISS@ and consistent design. In order to use {\i Synopse mORMot framework}, you should better be familiar with some of those definitions. First of all, a {\f1\fs20 @*Synopse.inc@} include file is provided, and appears in most of the framework units: !{$I Synopse.inc} // define HASINLINE USETYPEINFO CPU32 CPU64 It will define some conditionals, helping write portable and efficient code. In the following next paragraphs, we'll comment some main features of the lowest-level part of the framework, mainly located in @!Lib\SynCommons.pas@: - Unicode and @*UTF-8@; - {\f1\fs20 @*Currency@} type; - {\i @*Dynamic array@} wrappers ({\f1\fs20 TDynArray} and {\f1\fs20 TDynArrayHashed}); - {\f1\fs20 @*TDocVariant@} custom {\f1\fs20 variant} type for dynamic schema-less {\i object} or {\i array} storage. Other shared features available in {\f1\fs20 SynTests.pas} and {\f1\fs20 SynLog.pas} will be detailed later, i.e. @*Test@ing and @*Log@ging - see @12@. :32 Unicode and UTF-8 Our {\i mORMot} Framework has 100% UNICODE compatibility, that is compilation under {\i Delphi} 2009 and up (including latest {\i Delphi 10.3 Rio} revision). The code has been deeply rewritten and @*test@ed, in order to provide compatibility with the {\f1\fs20 String=UnicodeString} paradigm of these compilers. But the code will also handle safely Unicode for older versions, i.e. from {\i Delphi} 6 up to {\i Delphi} 2007. From its core to its uppermost features, our framework is natively @**UTF-8@, which is the de-facto character encoding for @*JSON@, {\i @*SQLite3@}, and most supported database engines. This allows our code to offer fast streaming/parsing in a @*SAX@-like mode, avoiding any conversion between encodings from the storage layer to your business logic. We also needed to establish a secure way to use strings, in order to handle all versions of {\i Delphi} (even pre-Unicode versions, especially the {\i Delphi} 7 version we like so much), and provide compatibility with the {\i @*FreePascal@ Compiler}. This consistency allows to circumvent any RTL bug or limitation, and ease long-term support of your project. Some string types have been defined, and used in the code for best cross-compiler efficiency: - {\f1\fs20 @**RawUTF8@} is used for every internal data usage, since both {\i SQLite3} and JSON do expect UTF-8 encoding; - {\f1\fs20 WinAnsiString} where {\i WinAnsi}-encoded {\f1\fs20 AnsiString} (code page 1252) are needed; - Generic {\f1\fs20 string} for {\i @*i18n@} (e.g. in unit {\f1\fs20 mORMoti18n}), i.e. text ready to be used within the VCL, as either {\f1\fs20 AnsiString} (for {\i Delphi} 2 to 2007) or {\f1\fs20 UnicodeString} (for {\i Delphi} 2009 and later); - {\f1\fs20 RawUnicode} in some technical places (e.g. direct Win32 *W() API call in {\i Delphi} 7) - note: this type is NOT compatible with {\i Delphi} 2009 and later {\f1\fs20 UnicodeString}; - {\f1\fs20 @**RawByteString@} for byte storage (e.g. for {\f1\fs20 FileFromString()} function); - {\f1\fs20 @**SynUnicode@} is the fastest available Unicode {\i native} string type, depending on the compiler used (i.e. {\f1\fs20 @**WideString@} before {\i Delphi} 2009, and {\f1\fs20 UnicodeString} since); - Some special conversion functions to be used for {\i Delphi} 2009+ {\f1 UnicodeString} (defined inside {\f1\fs20 \{$ifdef UNICODE\}...\{$endif\}} blocks); - Never use {\f1\fs20 AnsiString} directly, but one of the types above. Note that {\f1\fs20 RawUTF8} is the preferred {\f1\fs20 string} type to be used in our framework when defining textual properties in a {\f1\fs20 @*TSQLRecord@} and for all internal data processing. It is only when you're reaching the User Interface layer that you may convert explicitly the {\f1\fs20 RawUTF8} content into the generic VCL {\f1\fs20 string} type, using either the {\f1\fs20 Language. UTF8ToString} method (from {\f1\fs20 mORMoti18n.pas} unit) or the following function from {\f1\fs20 SynCommons.pas}: !/// convert any UTF-8 encoded String into a generic VCL Text !// - it's prefered to use TLanguageFile.UTF8ToString() in mORMoti18n.pas, !// which will handle full i18n of your application !// - it will work as is with Delphi 2009+ (direct unicode conversion) !// - under older version of Delphi (no unicode), it will use the !// current RTL codepage, as with WideString conversion (but without slow !// WideString usage) !function UTF8ToString(const Text: RawUTF8): string; Of course, the {\f1\fs20 StringToUTF8} method or function are available to send back some text to the @*ORM@ layer.\line A lot of dedicated conversion functions (including to/from numerical values) are included in {\f1\fs20 SynCommons.pas}. Those were optimized for speed and multi-thread capabilities, and to avoid implicit conversions involving a temporary {\f1\fs20 string} variable. Warning during the compilation process are not allowed, especially under Unicode version of {\i Delphi} (e.g. {\i Delphi} 2010): all string conversion from the types above are made explicitly in the framework's code, to avoid any unattended data loss. If you are using older version of Delphi, and have an existing code base involving a lot of {\f1\fs20 @*WideString@} variables, you may take a look at the {\f1\fs20 @*SynFastWideString.pas@} unit. Adding this unit in the top of your {\f1\fs20 .dpr} uses clauses will let all {\f1\fs20 WideString} process use the Delphi heap and its very efficient {\i @*FastMM4@} memory manager, instead of the much slower BSTR Windows API. Performance gain can be more than 50 times, if your existing code uses a lot of {\f1\fs20 WideString} variables. Note that using this unit will break the compatibility with BSTR/COM/OLE kind of string, so is not to be used with COM objects. In all cases, if you need {\i Unicode} support with older versions of Delphi, consider using our {\f1\fs20 RawUTF8} type instead, which is much better integrated with our framework, and has less overhead. :33 Currency handling Faster and safer way of comparing two {\f1\fs20 @*currency@} values is certainly to map the variables to their internal {\f1\fs20 Int64} binary representation, as such: !function CompCurrency(var A,B: currency): Int64; !var A64: Int64 absolute A; ! B64: Int64 absolute B; !begin ! result := A64-B64; !end; This will avoid any rounding error during comparison (working with *10000 integer values), and is likely to be faster than the default implementation, which uses the FPU (or SSE2 under {\i x64} architecture) instructions. Some direct {\f1\fs20 currency} processing is available in the {\f1\fs20 SynCommons.pas} unit. It will by-pass the FPU use, and is therefore very fast.\line There are some functions using the {\f1\fs20 Int64} binary representation (accessible either as {\f1\fs20 PInt64(@aCurrencyVar)^} or the {\f1\fs20 absolute} syntax): - {\f1\fs20 function Curr64ToString(Value: Int64): string;} - {\f1\fs20 function StrToCurr64(P: PUTF8Char): Int64;} - {\f1\fs20 function Curr64ToStr(Value: Int64): RawUTF8;} - {\f1\fs20 function Curr64ToPChar(Value: Int64; Dest: PUTF8Char): PtrInt;} - {\f1\fs20 function StrCurr64(P: PAnsiChar; const Value: Int64): PAnsiChar;} Using those functions can be {\i much} faster for textual conversion than using the standard {\f1\fs20 FloatToText()} implementation. They are validated with provided regression tests. Of course, in normal code, it is certainly not worth using the {\f1\fs20 Int64} binary representation of {\f1\fs20 currency}, but rely on the default compiler/RTL implementation. In all cases, having optimized functions was a need for both speed and accuracy of our ORM data processing, and also for @27@. Note that we discovered some issue in the FPC compiler, when {\f1\fs20 currency} is used when compiling from {\i x64-win64}: {\f1\fs20 currency} values comparison may be wrongly implemented using {\i x87} registers: we found out that using a {\i i386-win32} FPC compiler is a safer approach, even targetting {\i x64-win64} - at least for the trunk in 2019/11. \page :48 TDynArray dynamic array wrapper Version 1.13 of the {\f1\fs20 SynCommons.pas} unit introduced two kinds of wrapper: - Low-level @*RTTI@ functions for handling record types: {\f1\fs20 RecordEquals, RecordSave, RecordSaveLength, RecordLoad}; - {\f1\fs20 TDynArray} and {\f1\fs20 TDynArrayHashed} objects, which are wrappers around any {\i @*dynamic array@}. With {\f1\fs20 TDynArray}, you can access any {\i dynamic array} (like {\f1\fs20 TIntegerDynArray = array of integer}) using {\f1\fs20 TList}-like properties and methods, e.g. {\f1\fs20 Count, Add, Insert, Delete, Clear, IndexOf, Find, Sort} and some new methods like {\f1\fs20 LoadFromStream, SaveToStream, LoadFrom, SaveTo, Slice, Reverse,} and {\f1\fs20 AddArray}. It includes e.g. fast binary @*serialization@ of any {\i dynamic array}, even containing strings or records - a {\f1\fs20 CreateOrderedIndex} method is also available to create individual index according to the {\i dynamic array} content. You can also serialize the array content into @*JSON@, if you wish. One benefit of {\i dynamic arrays} is that they are reference-counted, so they do not need any {\f1\fs20 Create/try..finally...Free} code, and are well handled by the {\i Delphi} compiler. For performance-critical tasks, dynamic array access is very optimized, since its whole content will be allocated at once, therefore reducing the memory fragmentation and being much more CPU cache friendly. {\i Dynamic arrays} are no replacement to a {\f1\fs20 @*TCollection@} nor a {\f1\fs20 TList} (which are the standard and efficient way of storing class instances, and are also handled as @*published properties@ since revision 1.13 of the framework), but they are very handy way of having a list of content or a dictionary at hand, with no previous class nor properties definition. You can look at them like Python's list, tuples (via records handling) and dictionaries (via {\f1\fs20 Find} method, especially with the dedicated {\f1\fs20 TDynArrayHashed} wrapper), in pure {\i Delphi}. Our new methods (about searching and serialization) allow most usage of those script-level structures in your {\i Delphi} code. In order to handle {\i dynamic arrays} in our @*ORM@, some @*RTTI@-based structure were designed for this task. Since {\i dynamic array of records} should be necessary, some low-level fast access to the record content, using the common RTTI, has also been implemented (much faster than the "new" @*enhanced RTTI@ available since {\i Delphi} 2010). : TList-like properties Here is how you can have method-driven access to the {\i dynamic array}: !type ! TGroup: array of integer; !var ! Group: TGroup; ! GroupA: TDynArray; ! i, v: integer; !begin ! GroupA.Init(TypeInfo(TGroup),Group); // associate GroupA with Group ! for i := 0 to 1000 do ! begin ! v := i+1000; // need argument passed as a const variable ! GroupA.Add(v); ! end; ! v := 1500; ! if GroupA.IndexOf(v)<0 then // search by content ! ShowMessage('Error: 1500 not found!'); ! for i := GroupA.Count-1 downto 0 do ! if i and 3=0 then ! GroupA.Delete(i); // delete integer at index i !end; This {\f1\fs20 TDynArray} wrapper will work also with {\f1\fs20 array of string} or {\f1\fs20 array of record}... Records need only to be packed and have only not reference counted fields ({\f1\fs20 byte, integer, double}...) or {\f1\fs20 string} or {\f1\fs20 variant} reference-counted fields (there is no support of nested {\f1\fs20 Interface} yet). {\f1\fs20 TDynArray} is able to handle {\f1\fs20 record} within {\f1\fs20 record}, and even {\i dynamic arrays} within {\f1\fs20 record}. Yes, you read well: it will handle a {\i dynamic array} of {\f1\fs20 record}, in which you can put some {\f1\fs20 string} or whatever data you need. The {\f1\fs20 IndexOf()} method will search by content. That is e.g. for an {\f1\fs20 array of record}, all record fields content (including {\f1\fs20 string} properties) must match. Note that {\f1\fs20 TDynArray} is just a wrapper around an existing {\i dynamic array} variable. In the code above, {\f1\fs20 Add} and {\f1\fs20 Delete} methods are modifying the content of the {\f1\fs20 Group} variable. You can therefore initialize a {\f1\fs20 TDynArray} wrapper on need, to access more efficiently any native {\i Delphi} {\i dynamic array}. {\f1\fs20 TDynArray} doesn't contain any data: the elements are stored in the {\i dynamic array} variable, not in the {\f1\fs20 TDynArray} instance. : Enhanced features Some methods were defined in the {\f1\fs20 TDynArray} wrapper, which are not available in a plain {\f1\fs20 TList} - with those methods, we come closer to some native generics implementation: - Now you can save and load a {\i dynamic array} content to or from a stream or a string (using {\f1\fs20 LoadFromStream/SaveToStream} or {\f1\fs20 LoadFrom/SaveTo} methods) - it will use a proprietary but very fast binary stream layout; - And you can sort the {\i dynamic array} content by two means: either {\i in-place} (i.e. the array elements content is exchanged - use the {\f1\fs20 Sort} method in this case) or via an external integer {\i index look-up array} (using the {\f1\fs20 CreateOrderedIndex} method - in this case, you can have several orders to the same data); - You can specify any custom comparison function, and there is a new {\f1\fs20 Find} method will can use fast binary search if available. Here is how those new methods work: !var ! Test: RawByteString; !... ! Test := GroupA.SaveTo; ! GroupA.Clear; ! GroupA.LoadFrom(Test); ! GroupA.Compare := SortDynArrayInteger; ! GroupA.Sort; ! for i := 1 to GroupA.Count-1 do ! if Group[i]T