Hi all,

I think I've finally figured out the solution to at least a couple of
the significant internal design problems, and I'd like some feedback
on my ideas.  I'm not in a hurry to implement the solutions, and I've
got other development priorities that will likely overshadow this for
a while (in particular, replacing webrick and working on PuppetShow),
but I think solving these problems is pretty important.  I took a
crack at them in the fall, but it was too much work in the time I had
allotted.

Here are the changes I'm thinking of:

First, make 'retrieve' (on both attributes and types) return a value,
rather than setting <at> is internally.  This makes the classes much more
generically useful, and it means I can get rid of all of the code
related to caching or flushing those values.  The values would get
stored in the transaction, and each transaction would have a new
value list.

As part of this change, the [] methods would be changed to directly
modify and retrieve current state.  Right now, they set the <at> should
values and retrieve the ' <at> is' values, which is really confusing, and
probably just plain stupid.  I've actually stopped using those
methods internally because I was getting them wrong so often
(instead, I use 'is' and 'should' methods).

Second, change 'insync?' and 'sync' to accept values.  Thus, the
StateChange instances would pull the values using 'retrieve', and
then pass them back into the objects when necessary.  This makes the
transactions and StateChanges more powerful controllers, rather than
just triggering changes in internal state.

Third, create a cache in each transaction to store the current and
desired values.  The cache would also store the results of the
'insync' tests, so that the transaction could report exactly which
states weren't in sync.  This information is kind of available using
the statechange instances, but it's not very accessible.  This change
might actually make the StateChange class either unnecessary, or at
least less important.

Fourth, provide some loose hooks between the resources and this
transaction cache, so that resources know when they're running inside
of a transaction.

The most important hook would be that the resources would know how to
look up other resources from the transaction.  Right now, all of the
resource classes have [] methods, and they all place the same
restrictions in memory that the language places:  You can't have more
than one instance of an object that manages a given resource.  This
restriction makes sense in the language, but I don't think it makes
sense in the library, especially since it's actually a real pain to
manage creation and deletion of resources.  I have a lot of code that
removes resources after every operation because of this restriction,
and the only reason the restriction even exists is because I was
confused when I did the initial implemention; the restriction should
only ever have been in the language and never in the library.

Another important hook would be the ability to cache random
information in the transaction.  For instance, a file resource could
cache its 'stat' object in the transaction, so that every transaction
forced the resource to get a new stat.  This, again, would simplify a
good bit of internal code, and it would reduce the likelihood of
failures related to caching problems.

The damage:

All existing types would have to be migrated to this new system.
I've done 9/10s of the migration in a previous branch, but I had to
give up because I was on a release deadline and the 'cron' type (the
bane of my existence) was impossible to migrate.  I would need to
modify 'cron' to use providers, which would allow me to fix the
problems that stopped the work before.

I think I can get backwards compatibility for most types until the
migration is complete.  There are accessors for all of the variables
in question, so I can check the work and throw warnings when things
don't behave as they should yet still function correctly.

Also, it's going to take a while to do this.  I don't really know how
long, but I'd expect at least 3-4 days, plus 1-2 days fixing the cron
type (did I mention that it's my bane?).  The cron type needs to be
fixed anyway, though.

The results:

First and foremost, types would no longer have to maintain internal
state, which is a pretty big deal both in terms of reducing code
quantity and failures.

Second, the resources would become much more useful outside of
transactions.  Currently, if you want to figure out a file's uid, you
have to create the file, call 'retrieve' on it, then call 'file.is
(:owner)', which is just silly.  This way, you could just run 'file
[:owner]' and get the value right then.  This is an important fact
for me, because it suddenly becomes a lot easier for other ruby
applications to use the Puppet library to interact with systems,
whereas they'd currently pretty much have to follow Puppet's
transactional model to get anything useful done.

Third, it'd be much easier to create far better reports, because the
transaction would have easy access to all resources, and the current
vs. desired state of all of their attributes.

Comments?  Questions?  Recommendations?

Hi all,

I think I've finally figured out the solution to at least a couple of
the significant internal design problems, and I'd like some feedback
on my ideas.  I'm not in a hurry to implement the solutions, and I've
got other development priorities that will likely overshadow this for
a while (in particular, replacing webrick and working on PuppetShow),
but I think solving these problems is pretty important.  I took a
crack at them in the fall, but it was too much work in the time I had
allotted.

Here are the changes I'm thinking of:

First, make 'retrieve' (on both attributes and types) return a value,
rather than setting <at> is internally.  This makes the classes much more
generically useful, and it means I can get rid of all of the code
related to caching or flushing those values.  The values would get
stored in the transaction, and each transaction would have a new
value list.

As part of this change, the [] methods would be changed to directly
modify and retrieve current state.  Right now, they set the <at> should
values and retrieve the ' <at> is' values, which is really confusing, and
probably just plain stupid.  I've actually stopped using those
methods internally because I was getting them wrong so often
(instead, I use 'is' and 'should' methods).

Second, change 'insync?' and 'sync' to accept values.  Thus, the
StateChange instances would pull the values using 'retrieve', and
then pass them back into the objects when necessary.  This makes the
transactions and StateChanges more powerful controllers, rather than
just triggering changes in internal state.

Third, create a cache in each transaction to store the current and
desired values.  The cache would also store the results of the
'insync' tests, so that the transaction could report exactly which
states weren't in sync.  This information is kind of available using
the statechange instances, but it's not very accessible.  This change
might actually make the StateChange class either unnecessary, or at
least less important.

Fourth, provide some loose hooks between the resources and this
transaction cache, so that resources know when they're running inside
of a transaction.

The most important hook would be that the resources would know how to
look up other resources from the transaction.  Right now, all of the
resource classes have [] methods, and they all place the same
restrictions in memory that the language places:  You can't have more
than one instance of an object that manages a given resource.  This
restriction makes sense in the language, but I don't think it makes
sense in the library, especially since it's actually a real pain to
manage creation and deletion of resources.  I have a lot of code that
removes resources after every operation because of this restriction,
and the only reason the restriction even exists is because I was
confused when I did the initial implemention; the restriction should
only ever have been in the language and never in the library.

Another important hook would be the ability to cache random
information in the transaction.  For instance, a file resource could
cache its 'stat' object in the transaction, so that every transaction
forced the resource to get a new stat.  This, again, would simplify a
good bit of internal code, and it would reduce the likelihood of
failures related to caching problems.

The damage:

All existing types would have to be migrated to this new system.
I've done 9/10s of the migration in a previous branch, but I had to
give up because I was on a release deadline and the 'cron' type (the
bane of my existence) was impossible to migrate.  I would need to
modify 'cron' to use providers, which would allow me to fix the
problems that stopped the work before.

I think I can get backwards compatibility for most types until the
migration is complete.  There are accessors for all of the variables
in question, so I can check the work and throw warnings when things
don't behave as they should yet still function correctly.

Also, it's going to take a while to do this.  I don't really know how
long, but I'd expect at least 3-4 days, plus 1-2 days fixing the cron
type (did I mention that it's my bane?).  The cron type needs to be
fixed anyway, though.

The results:

First and foremost, types would no longer have to maintain internal
state, which is a pretty big deal both in terms of reducing code
quantity and failures.

Second, the resources would become much more useful outside of
transactions.  Currently, if you want to figure out a file's uid, you
have to create the file, call 'retrieve' on it, then call 'file.is
(:owner)', which is just silly.  This way, you could just run 'file
[:owner]' and get the value right then.  This is an important fact
for me, because it suddenly becomes a lot easier for other ruby
applications to use the Puppet library to interact with systems,
whereas they'd currently pretty much have to follow Puppet's
transactional model to get anything useful done.

Third, it'd be much easier to create far better reports, because the
transaction would have easy access to all resources, and the current
vs. desired state of all of their attributes.

Comments?  Questions?  Recommendations?