Eidetic Development Objectives

Not _necessarily_ in the following order, but perhaps roughly in this order depending on needs.

1. (DONE) Extensible module system
2. (DONE) Replace template system with Template Toolkit
3. (DONE) Packaging
4. (DONE) Move to Object-oriented architecture
5. (DONE) Reunified forks
6. (DONE) Extensible queries
7. Extensible command framework
8. Extensible output framework
9. Daemonization capability
10. Extensible widgeting system
11. Database schema management
12. Templatified emails
13. Enhanced use of TT2
14. Email interface

Early in the Eidetic 1.x series, the codebase grew difficult to install due to its complexities, due mostly to my unfamiliarity with Perl standards for config files, make-based install, etc.

A major focus through 2003 was to gain competence in the appropriate packaging techniques. Much of this effort was conducted as part of the DocSys development work, from which the current Eidetic 2.000 has been derived.

With the release of Eidetic 2.000 we are confident that this objective is met, though we expect to remain dedicated to making the packaging as good as can be.

Due to Eidetic's lack of an extensible command framework (see below), when new features were needed we folded them into the core. In fact, we didn't really distinguish between the core lib and the application (not that we didn't want to, just that it was challenging to put in that level of abstraction without losing the flexibility and power of being able to hardcode stuff).

One consequence has been the emergence of a few 'forks' of the codebase, customized for specific applications - STP, DocSys, etc. A major objective for the 2.000 release is to establish Eidetic as a common lib shared by all these apps and eliminate the need for the forking approach. It isn't as clean or well-organized as would be desired, but it's at least into a single codebase.

Eidetic was originally developed under Perl 4.x, which didn't have good support for OO syntax; while it's been rewritten several times since then, it was slow to move away from its functional architecture into an OO one.

One reason for this was that the problem it attempted to address - wrappering relational database tables - is not really object oriented. The amount of interconnectivity between tables is crosspurpose to object oriented design, which likes to segment things into independent units.

However, from a packaging standpoint, encapsulation is beneficial in and of itself, as it makes abstraction and reuse more feasible. This approach will eventually (see below) allow 'subclassing' to add business logic, so that default commands, queries and handlers can be overridden when one needs to do custom handling for a particular application.

This makes the codebase a bit better organized and encapsulates data a bit more safely, and lays a foundation that'll allow breaking out the custom bits into independent modules.

TODO

One of the major issues we've run into is that for many web applications is that while Eidetic's default commands are great for getting the web app going, there are invariably feature requests from users that require adding specialized 'business logic'. To date we've been embedding this into the Eidetic codebase itself, but cringe every time since it bloats the code with useless hacks and such, and over time has made the poor thing warty and funky.

What we need is an extensible command framework that'll permit adding business logic externally to the core. The approach we're considering is having a core set of default commands that can be subclassed at the application level as needed. This is a common architectural solution for these types of problems, and is fairly well supported in terms of language structure so seems like the logical way to go.

Currently Eidetic applications can be run as cgi's or to a limited extent as commandline utilities. We anticipate that being able to run Eidetic apps as daemonized servers will provide several benefits.

First, this provides a better way to handle authorization control for commandline apps; cgi programs are able to leverage the Apache web server for authorization, but it's trickier at the commandline level. Being able to run the service as a daemon allows us a wider degree of freedom and flexibility for handling authorization.

Second, it may benefit cgi applications by being able to move out stateful stuff into the daemon. Currently, the cgi versions of eidetic have high startup overhead that must be gone through for every single operation, such as loading up the authorized user's preferences, table structures, etc. In a daemon this information can be kept in memory for a while so that repeated commands can reuse the same info. This is an alternative approach to using mod_perl.

Third, it allows unified service handling in an interface-neutral manner. For instance, a document management system daemon could be run on a server and accessed by a web front end, a commandline tool, GUI's, and perhaps even mounted as a filesystem in its own right.

For Eidetic's web interface, the developer can design forms for input and editing of data, but these must be manually designed. Eidetic also has a routine for automatically generating these forms but they don't look that pretty and aren't very type-specific; for instance it isn't smart enough to know when it should use dropdowns or file-selectors. Further, this automatic generator is highly specific to HTML.

A better approach would be to generalize the concept of user interface internally, to the point that the autogenerator could simply communicate the user interface element types and let the client determine the precise widgets to use. This is akin to the XUL concept and might be compatible with it. This would also include a way to define a widget as an aggregate, such that you could have a dropdown with an 'Other...' field that allows input into an associated input box.

These widgets would be defined not only by the type(s) of UI elements to use but also the validation code. Input validation can be a substantial piece of the puzzle for web applications since bare HTML is so primitive as an interface language; one must build specialized javascript into the forms (and update as needed as things change), create special error screens, and do internal validation as well (in case the user turns javascript off). This system would provide a unified framework for handling validation, such that each widget would have associated with it the appropriate validation mechanisms - javascript, regexps, or perl code - that can be loaded and run by the appropriate code.

The widget definitions are loaded from the database instead of being part of the codebase. The reason for this is to enable an extensibility system, so one can add functionality without having to upgrade or modify the codebase. Often the definitions are highly application specific, so you wouldn't want them in the codebase anyway.

Eidetic's core mission in life is to make it easy to create even the trivialist web database applications. In its original incarnation it was built atop Berkeley DB, and handled table layout and definition internally; this was powerful in that it allowed defining and extending table definitions on the fly.

Moving to SQL with Eidetic 1.0 brought with it the limitation that table definition had to be handled by a database administrator. This effectively eliminated the ability to do database schema management on the fly.

However, databases do provide mechanisms for redefining tables; it's just more complex. What we want to do is represent the db schema in the database, so that we can use Eidetic operations to edit itself. New fields could be added or removed, or their definitions changed.

An advantage of this is allows a more highly abstracted way to associate data with field types. Basic SQL (at least, as implemented by mysql) doesn't really have an extensible typing system; if you need a complex type you have to serialize it into one of the primitive types (blob, text, double) or involve use of subtables and the additional logic to manage them.

What we want is to be able to express data elements with higher level typing, such as "ZipCode" or "MailingAddress". By attaching a data type object to a field in this way, we can associate validation or display logic (see Extensible widget system) as "part" of the type, instead of having to keep track of it at the application logic layer.

Invariably an application - especially a web app - needs to send stuff through email. This could be sending a receipt to a user or notifying an operator of actions that need to be taken. However, handling email notices is haphazard currently.

What we need is to employ templates for emails much as we do for the web pages, so that administrators can easily customize the text of their service notices without having to modify the codebase. They might wish to manage them in a content management system or with a simple web-based editor like vwebedit.

Eidetic was originally developed with its own internal template system. It was refactored to use Template-Toolkit instead, although some reminants of the old system are still present in certain corners. We anticipate migrating these remaining bits off into TT2 macros and processors, but we also recognize that Template Toolkit has power well beyond what we've done with it.

Using the application with web, commandline, and daemon interfaces is fine for almost everything, but there are some situations where people prefer being able to do things via email. No specific applications are currently anticipated, so this is left as lowest priority.