[Camps-users] Upcoming development session and a proposed way forward

Jon Jensen jon at endpoint.com
Mon Jan 12 23:52:22 UTC 2009


On Sun, 11 Jan 2009, Ethan Rowe wrote:

> No separation of "hosts", "services", and "resources".  Just 
> "resources".
>
> A resource may be a host, or a service.  There's no need for 
> distinguishing them.
>
> One resource can depend on another.
>
> One resource may contain another.

That all makes sense, if it can be implemented to be relatively easy to 
use.

How would plain files fit in? Or NFS-mounted files (which are used locally 
but served remotely)?

> A resource has two basic "command interfaces", by which I mean ways of
> issuing commands/requests to the thing represented by the resource (not
> "interfaces" in the sense of Java-esque interfaces that a class implements):
> * a "system" interface: this is the interface through which the camp
> system issues commands/requests to initialize and configure the resource
> in question (i.e. if you need to initialize a new Postgres cluster, you
> use the Postgres resource's system interface)
> * a "service" interface: this is the interface that issues
> commands/requests to the actual public-facing interface offered by the
> resource in question; once you've configured your Postgres resource
> through its system interface, you issue DDL commands through its service
> interface.

I like this, as it would eliminate or at least encapsulate the problem 
we've had in the past with things designed to work locally (Perl 
file-manipulation calls) suddenly needing to be re-implemented to run over 
ssh on a remote server.

> Resources would have attributes that must be configured.  Attribute 
> values should generally be calculated algorithmically.  The current camp 
> system basically treats attribute calculation as a set of mathematical 
> functions (that don't look terribly mathematical); for a given camp 
> number X, each attribute has only one possible value f(X) (i.e. the camp 
> number determines everything, from paths to port numbers to hostnames).
>
> However, it should be possible for attributes to be persistent, meaning 
> that once derived they are permanently stored as part of a camp's 
> configuration, rather than dynamically calculated for all time.
>
> It must be possible to reinitialize or reconfigure a resource (or set of 
> resources), and persistent attributes can be preserved rather than 
> recalculated.  Control over what persistent attributes to blow away at 
> re-initialization time must be built in from the beginning if this is to 
> make sense.
>
> Resource configuration, once determined, should be serialized down to 
> some storage format like YAML, JSON, XML (!!!!!!!!!!!), etc., so that 
> persistent attributes are preserved.
>
> Furthermore, because the central camp system needs to know all 
> attributes of the extant camps, this configuration information must be 
> preserved in the central system.  However, to potentially allow for a 
> distributed camp system where camps are spread across N nodes in a 
> server cluster (yes, we have a camp system for which this would be 
> highly relevant), and simply to allow local inspection of the 
> configuration, each camp's resource configuration information should 
> persist within the camp itself (i.e. in userland rather than base 
> camp-systemland).
>
> This means having some configuration data scattered around, which is 
> kind of a drag.  But it's a manageable drag.  Camp commands for 
> manipulating configuration values (camp config set <name> <value>) could 
> only be counted as successful if they appear to work centrally and 
> locally, for instance.

This makes sense, and as much as I'm loath to give up the simple 
calculated values, we've needed something more complex in several cases, 
so it makes sense.

I'll note that not only do we need to store the calculated values, but 
they need to be in a kind of directory that shows them to be "taken", so 
they can't be used for something else, and there should be a kind of 
priority that keeps them sane.

For example, in your example of someone needing multiple memcached daemons 
in a camp, it'd be nice to have those extra ones' port numbers allocated 
from some extra pool outside the normal range calculated based on camp 
number, e.g. memcached #1 gets 8800 * $camp_number, but memcached #2-5 get 
the next available ports above 13000, or some such. That way most numbers 
will still be easy to remember (easy to spot mistakes), while more complex 
things are still possible. Does that make sense?

> I'd like to propose that individual camps be treated as resources as 
> well.  This is a fairly new idea (in my mind, anyway) and may be 
> completely ridiculous.  I'm just putting it out there.  Representing the 
> camps themselves as resources, and then the attributes of the camps 
> (i.e. numbers, owners, base paths, etc.) are managed like any other 
> resource attribute.

I like that too. Again, as long as it's not a huge pain to deal with.

> We do not fret about getting too generic, at least at first.  I.e. no 
> "Resource::Database" from which "Resource::Database::Postgres" and 
> "Resource::Database::MySQL" derive.  We implement postgres and mysql 
> independently, and if common things can obviously be factored out into 
> some common ancestor, we do it.

I encourage that. It'll reduce the tendency to try to force things to be 
similar in implementation when they may only be similar in external 
nomenclature. And as you say, they can be refactored later if it's a clear 
win.

To avoid the problem of vaporware discussions, I'd suggest you plan out 
the most profitable subset of all that as a prototype to build before & 
during the hackathon, and let's see how that goes before worrying too much 
more about it. Do you have enough to go on?

Thanks,
Jon

--
Jon Jensen
End Point Corporation
http://www.endpoint.com/


More information about the Camps-users mailing list