[Camps-users] Two tracks of camps system development

[Ethan Rowe]

Jon Jensen wrote:
[snip]
> Ethan and I have several ideas for camps "version 4" that would 
> greatly benefit one of our clients, and which we're discussing 
> implementing for them. I don't want us to do other work on version 4 
> until, hopefully, that client has us do the work. Ethan, would you 
> fill us all in on what you've discussed with them so far?

Happily.

The new design plans address a few critical areas where "version 3" 
falls down:
* extensibility: introduction of support for new services and the ease 
with which those services can be integrated into the system;
* configurability: improve the flexibility of configuration, such that a 
given camp type declares what services exist, their dependency order, 
etc. (this is in contrast to configuration options that give you some 
flexibility in a camp type so long as the application you're 
representing fits neatly into the 
one-database-with-one-appserver-with-one-webserver model);
* host support/awareness: introduce representations of different hosts 
(localhost versus any number of remote hosts) and provide the ability to 
control services on any of those hosts from the primary camp machine;
* unlimited service instances: enable the system to support N instances 
of any given service, within a single host or on different hosts (for 
instance, multiple Postgres clusters for a given camp, so database 
replication concerns can be addressed in the development environment; 
this is also potentially useful for service-oriented architecture, where 
multiple instances of the same app server may be necessary, with 
different configuration for each)

Additional plans that are less fundamental than those listed above:
* command unification: all commands for working with a camp will be 
unified within a master "camp" command, and the interface of each will 
be made consistent;
* administrative commands: introduce new commands so an administrator 
can perform some of the basic setup of a new camp type through 
command-line utilities rather than copying files by hand;
* improved ease of installation: the camp system itself should be 
considerably easier to install on a machine, via at least one of CPAN, 
RPM, etc.;
* programmatic interface: every critical command-line utility will have 
a corresponding method on a master Perl object, which should allow 
easier building of new, higher-level command-line utilities over time, 
and just makes the ways in which camps are used/controlled more flexible

We'll naturally use OOP for this rather than library/exporter modules.  
While keeping the Perl dependencies of the camp system quite light is 
desirable, we'll use Moose as the base object system; while I started 
some prototyping of a simple custom base object, thinking that Moose is 
ultimately more dependent-rich than we would want, Brian Miller 
helpfully pointed out that using Moose would be simpler as a start, and 
if the dependencies were a problem then we could extract Moose out of 
the base object over time.  Jon and I ultimately agreed with that line 
of thinking.

Here's a rough class hierarchy (these package names aren't final by any 
means):
* Camp::Base (bad name) -- base object from which all other objects are 
derived; uses Moose
* Camp::Master -- top level object that coordinates everything
** interacts with the master camp database, which is only used by 
Camp::Master;
*** master camp database: database (Postgres, MySQL, etc.), sqlite, or 
YAML+locking?
** password generation
* Camp::Service -- base class for services, which model pieces of 
software that can be deployed within a camp
** abstract classes of services:
*** databases: Postgres, MySQL, ...
*** vcs: Git, CVS, Subversion, Subversion+SVK, ...
*** static files: non-versioned htdocs, symlinks, ...
*** app servers: Interchange, Rails, ...
*** web servers: Apache, ...
** service attributes:
*** base path
*** port management using "port range" objects
*** socket file paths
* Camp::PortRange --  an object for representing the port needs of a 
given service:
** base port
** number of ports per camp
** initially, port determinations will be algorithmic, based on the base 
port number, the number of ports per camp, and the number of the camp in 
question
* Camp::Host -- object for representing an independent host in which 
services may be deployed
** localhost by default
** other hosts by configuration, accessible via service-specific 
protocol or ssh
* Camp::Resource -- models the actual stuff that goes into a camp, 
combining a service with a host
** resource attributes:
*** name: identifies the resource within the camp type; for simple camp 
types, defaulting to the service name would suffice
*** service: an instance of a Camp::Service subclass
*** host: an instance of Camp::Host (or a subclass), which defaults to 
localhost
** the resource has the opportunity to override the service ports, 
paths, etc. as needed.
* All objects should inherit common logic for configuration file 
parsing, and should know where to find their configuration files
** In reading configuration files, objects will share the following 
behavior:
*** any configuration file token that maps to a known attribute in the 
object in question will be assigned to that attribute, meaning that the 
value is subject to type constraints, validation, etc.
*** for configuration file tokens that don't map to a known attribute in 
the object, a generic attribute of that name will be autovivified, 
without any validation logic whatsoever
*** this allows us to define strict attributes with constraints on them 
while still maintaining the flexibility that comes from letting 
configuration files define arbitrary name/value pairs to be used through 
an application
*** this also means that reading a configuration file means that we're 
potentially extending a module rather than merely putting stuff into an 
object -- this magic will need some planning to make sure namespaces are 
managed effectively and predictably
* All objects will know how to render templates, knowing where the 
templates come from and where they go in the target camp

The configuration scheme will need to be significantly altered to 
accommodate these changes.  So, our thinking here is:
* All configuration files will use YAML; it's simple, flexible, 
powerful, etc.;
* configuration values will be subject to Text::ScriptTemplate 
evaluation for ASP-style embedded calculations; this is more verbose 
than the token-substitution method used in version 3, but it solves the 
0-n problem and ultimately balances an increase in verbosity with an 
enormous leap in flexibility/power;
** within Text::ScriptTemplate evaluations:
*** marshal in a $master object (the Camp::Master instance, through 
which all camp resources can be accessed);
*** marshal in a $self object (which is a reference to the Camp::Host, 
Camp::Service, or Camp::Resource being configured);
** the templates for rendering into a camp will also use 
Text::ScriptTemplate;
* within the "global" camp space, we'll have system-wide configuration 
information for services;
* within the "camp type" space, we'll have configuration information to 
extend/define:
** hosts;
** services;
** resources;
** state the dependency order between resources;
** define the resources that are default for making a new camp, 
stopping/(re)starting a camp, etc.;
* within an individual camp, we'll have configuration information to 
extend/define:
** hosts;
** services;
** resources;
** list the resources used within the camp

The configuration layout would be something like:

<CAMPROOT>/
  services/
    <SERVICE_NAME>/
      config.yml
      etc/ (where templates to be rendered live)
  <TYPE>/
    config.yml (default resources for start/stop/restart/make; specify 
dependency order)
    resources/
      <RESOURCE_NAME>/
        config.yml
        etc/


This design lays the foundation for a very extensible, flexible system 
that could be used in all sorts of development situations, not just its 
initial space of webapps.  The management of hosts and combination of 
service and host into a named resource gives some essential tools for 
representing environments of arbitrary complexity.  The host design can 
presumably be used in the not-too-distant future to enable the use of 
virtual machines, such that making a new camp means auto-instantiating a 
new virtual machine from an existing image.  But we don't need to worry 
about that right now; looking at the body of the email I just wrote, I'd 
say we have enough to worry about for the present.  :)

Thanks.
- Ethan

-- 
Ethan Rowe
End Point Corporation
ethan at endpoint.com