headers
Ethan Jucovy | 10 Sep 16:40
Picon

Moving more stuff to github, (temporarily?) shutting down SVN and Trac

Hey,


I'm going to be shifting servers for a bunch of stuff over the next few weeks, mainly in preparation for moving CoActivate.org to a new hosting provider.  One of the first steps I'd like to take is shutting down the Slicehost server that's currently hosting the SVN and Trac sites for OpenCore & friends.  I'll probably set them up again once all the new servers are in place, but for the moment it'll be much more convenient for me to just keep them archived on my laptop, until everything has settled.

So, my hazy plan, which I'd like to at least start moving on this weekend, is something like this:

1) Finish moving all active projects from https://svn.socialplanning.org/svn to https://github.com/socialplanning, including the opencore repository itself.

2) Make sure opencore builds work properly from Github.

3) Take https://svn.socialplanning.org/svn offline and copy its full repository to my laptop.

4) Make sure opencore builds still work properly from Github with SVN totally offline.

5) Release a long-overdue OpenCore 0.19.0 since I'll've just done all the necessary gruntwork for a release anyway.


7) For each of the six Trac sites there, dump the database, archive the project, and copy them to my laptop.

8) Make sure I can run the SVN and Tracs from my laptop, to double check that I didn't miss anything.

9) Dig around on the trac+svn server to make sure there's nothing else important on it, and then shut it down entirely.

Let me know ASAP if this is a problem for you.

-Ethan


--
Archive: http://www.coactivate.org/[…]/1315665649356
To unsubscribe send an email with subject "unsubscribe" to opencore-dev <at> lists.coactivate.org. Please contact opencore-dev-manager <at> lists.coactivate.org for questions.
Permalink | Reply | 3 comments |
headers
Ethan Jucovy | 7 Jun 17:16
Picon

dist.socialplanning.org now on github

Just a heads up -- I've moved http://dist.socialplanning.org from my Slicehost account to a Github Pages site.  Hopefully this will make builds faster, and be more maintainable.  And save me space. :)


The git repo that's being served lives here: https://github.com/socialplanning/dist.socialplanning.org

There's a rebuild_egg_index.py script there which should be re-run whenever a new file is added to the repo -- since Github Pages doesn't serve Apache-style dynamic index pages if there's no index.html file.

The repo is over the Github soft size limit for an account (0.3G; it's closer to 0.9G) but hopefully they won't mind; Github staff have said in various forums that they don't worry much about the size limit for free software projects.

Also, if you want to be added to the socialplanning Github Organization, just ping me or rmarianski.

-Ethan


--
Archive: http://www.coactivate.org/[…]/1307459829237
To unsubscribe send an email with subject "unsubscribe" to opencore-dev <at> lists.coactivate.org. Please contact opencore-dev-manager <at> lists.coactivate.org for questions.
Permalink | Reply | 1 comment |
headers
elf Pavlik | 16 May 01:43
Gravatar

Invitation: Federated Social Web - Conference Europe & W3C Group

Hello,

I would like to invite you to join us for the upcoming Federated Social Web Europe Conference - June 3-5th in Berlin
http://d-cent.org/fsw2011/

And also consider joining W3C Federated Social Web incubator group (soon changing into community group)
http://www.w3.org/2005/Incubator/federatedsocialweb/

We send invites to various projects working on protocols for federated social networking and developers
of open source platforms implementing them. You can see growing list of invites here:
http://www.w3.org/2005/Incubator/federatedsocialweb/wiki/FSWE2011_-_Sent_Invitations

If you would like to join us for the conference but need help with organizing your travel and stay in Berlin,
please add yourself to the list on this page:
http://www.w3.org/2005/Incubator/federatedsocialweb/wiki/FSWE2011_-_Travel_Support_Requests

Please also feel invited to take a closer look on the wiki of our group, still in it's early stage but already
providing some information about group participants, related protocols and software - including list
of open source platforms where developers already work on implementing federation or consider
implementing it:
http://www.w3.org/2005/Incubator/federatedsocialweb/wiki/Main_Page

Last but not least, our public mailing list with its archives stays available here:
http://lists.w3.org/Archives/Public/public-xg-federatedsocialweb/

Looking forward to work together with you on improving interoperability of social networking platforms
and increasing freedom of all of us using them!

elf Pavlik
Participant of W3C Federated Social Web Incubator Group

--
Archive: http://www.coactivate.org/projects/opencore/lists/opencore-dev/archive/2011/05/1305503020187
To unsubscribe send an email with subject "unsubscribe" to opencore-dev <at> lists.coactivate.org.  Please
contact opencore-dev-manager@...e.org for questions.
Permalink | Reply | 0 comments |
headers
Ethan Jucovy | 23 Apr 16:00
Picon

Moving stuff to Github

I'm starting to move stuff to Github under an organization here: https://github.com/socialplanning -- starting with the smaller / more standaloney things from SVN.  Let me know if you want to be added to the organization.

So far I've moved:

* fassembler / opencore-fassembler_boot / opencore-fassembler_projects
* sven / svenweb
* libopencore
* flunc

Also Deliverance now lives here: https://github.com/deliverance/Deliverance/ (as of a few weeks now)

I don't have any plans to shut down the SVN repository, but I'd like to get actively maintained stuff over to Github.  I'm planning on keeping trac.socialplanning.org for issue/task tracking.

For posterity, here's how I'm porting the projects:

  $ git svn clone --stdlayout -A users.txt https://svn.socialplanning.org/svn/flunc /tmp/flunc.git
  $ cd /tmp/flunc.git
  $ cp -Rf .git/refs/remotes/tags/* .git/refs/tags/
  $ rm -Rf .git/refs/remotes/tags
  $ cp -Rf .git/refs/remotes/* .git/refs/
  $ rm -Rf .git/refs/remotes
  $ git branch -l  # check that it looks sane
  $ git tag -l # check that it looks sane
  $ git remote add origin git-9UaJU3cA/F/QT0dZR+AlfA@public.gmane.org:socialplanning/flunc.git
  $ git push --all
  $ git push --tags

If anyone wants the users.txt svn->git mapping file I'm using, let me know; I'll hold on to it.  You can use it to spam everybody who ever worked on OpenCore. :)



--
Archive: http://www.coactivate.org/[…]/1303567257368
To unsubscribe send an email with subject "unsubscribe" to opencore-dev <at> lists.coactivate.org. Please contact opencore-dev-manager <at> lists.coactivate.org for questions.
Permalink | Reply | 2 comments |
headers
Ethan Jucovy | 17 Dec 05:39
Picon

BZR/wiki explorations

On Sat, Jul 10, 2010 at 11:27 AM, Ethan Jucovy <ethan.jucovy-Re5JQEeQqe8@public.gmane.orgm> wrote:

As a test though I then removed portal_historiesstorage (IIRC - I think that's the one that physically stores the version objects) entirely from a test site and then repacked.  The database went from 34G to 4G.  So at least it's thoroughly confirmed now that old versions of binary content are the sole cause of ZODB bloat.

I'm going to keep investigating this.  One option I'm considering (not yet sure how seriously) is dumping CMFEditions and just keeping wiki history in a VCS repository (probably bazaar).  The opencore UI doesn't really support versioning of anything other than the body of a wikipage -- titles are technically versioned but I believe old page titles don't ever show up in the UI, and we don't version any complex objects -- so CMFEditions' object-versioning may just be unnecessary overkill anyway.  I'm already planning on writing code to transform a project wiki into a VCS repo for project export purposes, so it might make sense.  Curious if anyone feels otherwise. 

(I also noticed on plone-dev recently there was talk of replacing CMFEditions, and IIRC Martin Aspeli proposed a VCS backend.  This is by the way, but does make me think it's not crazy to jettison CMFEditions as a dependency.)

I checked in some code on trunk which converts a project wiki history to a bzr repository.  The immediate goal is just getting a complete history for project export, but if this continues to seem promising I'll be looking into using it for live backups as well.

Sorting all commits globally by date across a given project is important to me.  This is tricky, because CMFEditions storage is CVS-like (no global revision number) so an in-memory sort really doesn't work for large wikis.  Instead, I do the procedure in two phases: first, store all the commits' metadata in a sqlite db; then, retrieve them, sorted by timestamp, from the rdb and commit them to a bzr repo in order.  This is working surprisingly well.


I've tested it against some pretty difficult data from Coactivate: nearly up-to-date copies of the opencore project wiki; another large and long-running wiki; and a smaller/younger wiki with lots of non-ASCII text.  All are working well, after a few rounds of trial and error.[*]

Interesting stats:
 opencore wiki: 5551 checkins on 694 pages; 4.3M .bzr directory; 4.1M checkout
 another wiki: 5540 checkins on 193 pages; 6.7M .bzr directory; 4.2M checkout

Next steps:
 * Look into running this during project export, or as a separate project export action; maybe split it up into batches to be run by separate calls to the export queue
 * Look into implementing edit functionality by checking out a copy of the repo on edit and checking it back in after save

Later steps:
 * Look into providing the opencore wiki history / revert / version interfaces on top of a live BZR repo
 * Look into what interesting analyses can be done (just from casual glancing at `bzr log` output I'm seeing potentially interesting patterns)
 * Research bzr plugins

If I do move forward with this for a live wiki history backend, it will also be important to figure out how to cleanly uninstall CMFEditions and all its persistent objects -- which was my original motivation back in July.

[*] On Coactivate, some very old wiki pages have versions that used to be WickedDocs instead of our current object type.  These objects break the portal_repository storage - they're actually not even accessible on the web.  A patch to CMFEditions worked around this and allowed me to retrieve their contents and metadata :

Index: src/opencore-bundle/CMFEditions/StandardModifiers.py
===================================================================
--- src/opencore-bundle/CMFEditions/StandardModifiers.py (revision 62256)
+++ src/opencore-bundle/CMFEditions/StandardModifiers.py (working copy)
<at> <at> -350,7 +350,10 <at> <at>
             if attr_name not in clone_ids:
                 new_ob = getattr(obj, attr_name, None)
                 if new_ob is not None:
-                    repo_clone._setOb(attr_name, new_ob)
+                    try:
+                        repo_clone._setOb(attr_name, new_ob)
+                    except AttributeError:
+                        pass
 
         # Delete references that are no longer relevant
         for attr_name in clone_ids:



--
Archive: http://www.coactivate.org/[…]/1292560767291
To unsubscribe send an email with subject "unsubscribe" to opencore-dev <at> lists.coactivate.org. Please contact opencore-dev-manager <at> lists.coactivate.org for questions.
Permalink | Reply | 0 comments |
headers
Ethan Jucovy | 4 Dec 01:40
Picon

Do you use Listen 0.7 (trunk)?

Hi,


I'm wondering if there are many people using the Listen 0.7 series, which is Listen trunk and has had two PyPI releases (most recent: http://pypi.python.org/pypi/Products.listen/0.7.1 in Feb 2009)

The reason I ask is that I've been making changes on the 0.6 branch - mostly i18n bugfixes, but, more recently, an increasing number of other changes that aren't just one-line bugfixes.

I've been periodically making new bugfix releases of the Listen 0.6 series (most recent: http://pypi.python.org/pypi/Products.listen/0.6.4 in July 2010) but now that I'm making changes that are less like strict bugfixes, I'd much rather be making more "significant" releases.

Background: I've been changes to Listen as needed in OpenCore, the software that runs CoActivate.org and some other sites.  AFAIK the needs of OpenCore have generally driven Listen development for most of its history and the Listen committers have mostly been developers working on OpenCore.  Listen 0.7 provides a revamped user interface.  Unfortunately, the way that OpenCore uses Listen and overrides some of its views and templates makes it rather difficult to upgrade from 0.6 to 0.7 - precisely because of the UI changes in Listen 0.7.

So .. I'm not really sure how best to move forward here.  From the OpenCore perspective, the simplest thing to do is roll-back trunk to the 0.6 series and start calling releases 0.8.x .. which is more than a bit ugly, and its ugliness matters in proportion to the number of users that Listen has.  From the Listen perspective, the simplest thing to do is .. nothing, and force OpenCore to upgrade -- which has proven difficult the few times I've tried it -- and port its 0.6-series changes to trunk.  Another option is to splinter or fork Listen development - OpenCore's 0.6 version of Listen becomes some new product .. and development of Products.listen presumably stalls in its current state until somebody picks it up for a non-OpenCore use.

All of these are clearly suboptimal - so I'm wondering if anyone else has thoughts, vested interests, or recommendations.  If nobody does, I'll probably go for the fork/splinter option - because it demands the least work of me in OpenCore, and also doesn't make a mess of the book-keeping.

-Ethan


--
Archive: http://www.coactivate.org/[…]/1291423268164
To unsubscribe send an email with subject "unsubscribe" to opencore-dev <at> lists.coactivate.org. Please contact opencore-dev-manager <at> lists.coactivate.org for questions.
Permalink | Reply | 0 comments |
headers
Ethan Jucovy | 21 Nov 15:40
Picon

Two zcmlloader targets

A little while ago Paul checked in some changes to the zcml-autoinclusion stuff.  His checkins explain what's going on, but I wanted to mention it on the list too where it'll be easier to find later on.


Paul found that distribute and setuptools don't iterate over entrypoints in the same order.  The result was that if you install opencore with distribute, things break.  Some plugins (opencore.cabochon I think) were trying to use the IProvideSiteConfig utility during their own component initialization.  But the IProvideSiteConfig utility is itself registered by a plugin (fassembler.configparser) -- so there was an implicit ordering dependency, that fassembler.configparser's ZCML had to be loaded before opencore.cabochon's.  

The order in which plugins are loaded within a single <topp:includePlugins> directive is based on the order of items yielded from iter_entry_points.  With setuptools, fassembler.configparser's [topp.zcmlloader] entrypoint happened to appear first, but with distribute, it happened not to.  So there was a ComponentLookupError on startup if distribute was used.

The fix was to add a second plugin-point for ZCML: 

<topp:includePlugins target="opencore_siteconfig" />
<topp:includePlugins target="opencore" />

So, fassembler.configparser now registers itself as a plugin for "opencore_siteconfig", instead of "opencore", and it will always be loaded before other plugins.

Conceptually, this makes better sense: fassembler.configparser provides a utility whose existence is required by opencore, so it's a different kind of plugin than all the rest, which provide strictly optional extra features.  If we ever have any other things that fit this description, we should register them for "opencore_siteconfig" as well.

When I get around to porting opencore to z3c.autoinclude, this shouldn't change anything.  The feature (defining arbitrary plugin-points) exists there too.

The changesets here were 27774:27777.  That included one small change needed in ZCMLLoader itself, to make this feature work.


--
Archive: http://www.coactivate.org/[…]/1290350458585
To unsubscribe send an email with subject "unsubscribe" to opencore-dev <at> lists.coactivate.org. Please contact opencore-dev-manager <at> lists.coactivate.org for questions.
Permalink | Reply | 0 comments |
headers
Ethan Jucovy | 8 Nov 14:44
Picon

Build failures

The builds started failing on Friday -- the build itself is now broken (0), and also the Products.listen tests (1) and opencore tests (2) are failing.  Details:

0) The latest bundle gives a MaildropHost config.py with local modifications specific to Paul's computer.

This makes the build fail for totally stupid reasons: Fassembler doesn't replace the contents of the config.py file (in `fassembler maildrop`) until *after* it has started Zope/ZEO once (in `fassembler zeo`) so the file has to be importable without errors before the contents-swap.

This is causing these failures:


The hint is buried in the traceback:

IOError: [Errno 2] No such file or directory: '/usr/local/topp/team.openplans.org/builds/20090527/etc/opencore/maildrop/config.py'

Paul, could you revert that local modification in your bundle's maildrophost/config.py and upload a new bundle with the fix?

I'll look into rearranging the Fassembler tasks so that this doesn't have to matter.  (Also, I noticed that upgrading MaildropHost would give us a supported way to switch configs without hacking the source.  I filed a ticket to remember that: http://trac.socialplanning.org/opencore/ticket/2922)

---

1) Stray ``print`` in listen export breaks Products.listen doctests:


I assume the ``print`` statement is debug cruft, but I'm not 100% sure -- it was present in the long-standing trunk code that Paul backported:


---

2) Project export failing when success is expected:


I'm guessing this has to do with the additional blog failure modes added:


But I don't know why that would be -- I assume that test project should not have the blog featurelet installed, so the whole save_blogs method ought to exit early, before any failures could be triggered.


--
Archive: http://www.coactivate.org/[…]/1289223904154
To unsubscribe send an email with subject "unsubscribe" to opencore-dev <at> lists.coactivate.org. Please contact opencore-dev-manager <at> lists.coactivate.org for questions.
Permalink | Reply | 10 comments |
headers
Paul Winkler | 8 Nov 05:17
Favicon
Gravatar

Re: Opencore bundle updates

btw, the reason I did all that was to whip project export into better
shape.  I finally started using it "in anger" and ran into a bunch of
failures; all the problems I've hit so far are fixed now on opencore
trunk.  

Oh, and there's a script at
opencore/scripts/export_all_projects.py which you can try via `zopectl run`.

- PW

On Fri, Nov 05, 2010 at 09:00:59PM -0400, Paul Winkler wrote:
> I've just updated the "bundle" (the pile of Zope Products that
> fassembler puts in opencore/src/opencore-bundle), in order to get:
> 
> 1) some Listen archive export fixes
> 
> 2) a MailBoxer fix that allows it to work when the whole site is
> served via https
> 
> 3) some shell scripts that have long been in the bundle tarball but were
> never checked in to SVN.
> 
> Having done that: I'm not really active in opencore development
> anymore, except for the occasional little foray like this, but I gotta
> say, I think the workflow for keeping the "bundle" updated is
> terrible.  In order to do make these updates you have to
> 
> 1. make your updates or changes, and commit them - possibly in a
>   "vendor branch" (aka fork) if you don't have commit rights in the
>   original upstream repository.
> 
> 2. update the svn:externals property on the opencore-bundle directory
>   itself, and make sure you get the revision numbers right.
> 
> 3. remember to run the opencore-bundle/create-tarball.sh script
>    ... which only works if you have permission to upload stuff to
>    svn.openplans.org/eggs ...
>    ... and always bails out because MaildropHost/config.py always has
>    local changes because fassembler munges it on purpose, so you have
>    to temporarily revert that, then re-run the script, then put the
>    changes back in place.
> 
> If anybody stops after step 1, their local build now works as they
> intended but all future builds will still get some old, presumably
> broken code, in a place that's pretty obscure.  In my experience, this
> has happened a *lot*.
> 
> (I've just fixed the second problem with create-tarball.sh - it now
> ignores changes to MaildropHost/config.py, which should be safe as
> fassembler is always going to clobber that anyway.  But the rest of the
> issues remain.)
> 
> (There's also the problem that fassembler always gets the "current"
> plone3-compatible bundle, whatever that happens to be; there's no way
> for opencore-req.txt to specify anything more specific so you don't
> really know what you're getting or which opencore releases it'll
> actually work with.  i.e. this part of the build is totally *not*
> predictably repeatable)
> 
> Rant over!
> 
> P.S. Ethan, the svn:externals and script changes were applied to
> https://svn.openplans.org/svn/bundles/opencore-plone30 in revisions
> 27886:27897 .  You'll want to make the same changes on
> svn.coactivate.org.  Once that's done I can upload a new bundle to
> svn.openplans.org/eggs so at least it has the right svn url... but
> ideally fassembler and the create-tarball.sh script should be updated
> to stop looking for it on svn.openplans.org/eggs.
> 
> -- 
> 
> Paul Winkler, software engineer @ OpenPlans
> http://openplans.org/team/#paul-winkler
> irc.freenode.net: slinkp
> yahoo: slinkp23
> AIM:   slinkp1970
Permalink | Reply | 0 comments |
headers
Paul Winkler | 8 Nov 05:17
Favicon
Gravatar

Opencore bundle updates

I've just updated the "bundle" (the pile of Zope Products that
fassembler puts in opencore/src/opencore-bundle), in order to get:

1) some Listen archive export fixes

2) a MailBoxer fix that allows it to work when the whole site is
served via https

3) some shell scripts that have long been in the bundle tarball but were
never checked in to SVN.

Having done that: I'm not really active in opencore development
anymore, except for the occasional little foray like this, but I gotta
say, I think the workflow for keeping the "bundle" updated is
terrible.  In order to do make these updates you have to

1. make your updates or changes, and commit them - possibly in a
  "vendor branch" (aka fork) if you don't have commit rights in the
  original upstream repository.

2. update the svn:externals property on the opencore-bundle directory
  itself, and make sure you get the revision numbers right.

3. remember to run the opencore-bundle/create-tarball.sh script
   ... which only works if you have permission to upload stuff to
   svn.openplans.org/eggs ...
   ... and always bails out because MaildropHost/config.py always has
   local changes because fassembler munges it on purpose, so you have
   to temporarily revert that, then re-run the script, then put the
   changes back in place.

If anybody stops after step 1, their local build now works as they
intended but all future builds will still get some old, presumably
broken code, in a place that's pretty obscure.  In my experience, this
has happened a *lot*.

(I've just fixed the second problem with create-tarball.sh - it now
ignores changes to MaildropHost/config.py, which should be safe as
fassembler is always going to clobber that anyway.  But the rest of the
issues remain.)

(There's also the problem that fassembler always gets the "current"
plone3-compatible bundle, whatever that happens to be; there's no way
for opencore-req.txt to specify anything more specific so you don't
really know what you're getting or which opencore releases it'll
actually work with.  i.e. this part of the build is totally *not*
predictably repeatable)

Rant over!

P.S. Ethan, the svn:externals and script changes were applied to
https://svn.openplans.org/svn/bundles/opencore-plone30 in revisions
27886:27897 .  You'll want to make the same changes on
svn.coactivate.org.  Once that's done I can upload a new bundle to
svn.openplans.org/eggs so at least it has the right svn url... but
ideally fassembler and the create-tarball.sh script should be updated
to stop looking for it on svn.openplans.org/eggs.
Permalink | Reply | 0 comments |
headers
Ethan Jucovy | 4 Nov 18:07
Picon

New opencore buildbot running and green

I've set up a new buildbot here:



It's currently running builds of opencore-minimal with opencore trunk and the 0.18 branch, and opencore-maximal (wordpress, tasktracker, etc) with opencore trunk.  All builds run the flunc tests and the zopectl tests.

All the builds are now working, and are even running green!

I'm going to set up notifications soon.  I'm planning on sending build failure emails to the opencore-dev list.

I'll also be adding some more builds as I have time - I want to add a build for the precise code that Coactivate.org runs, and I'm open to adding builds for other sites' setups as well.  I'm also planning on setting up a Deliverance build on the same site.

There are a few tweaks I want to make to the builds themselves.  They don't destroy the MySQL databases between builds, so some of the setup for tasktracker/wordpress/etc isn't being fully tested.  But I think this is a good start.

The buildbot is itself built with fassembler using Paul's `fassembler buildmaster` and `fassembler buildslave` projects.

If anybody wants to run slaves, I'd be happy to offload some of the processing, just let me know and we can figure out how to set that up.

-Ethan


--
Archive: http://www.coactivate.org/[…]/1288890454249
To unsubscribe send an email with subject "unsubscribe" to opencore-dev <at> lists.coactivate.org. Please contact opencore-dev-manager <at> lists.coactivate.org for questions.
Permalink | Reply | 0 comments |

Gmane