Hi Alberto,

hm, never spotted such behavior. Check the exported archive for actual files, to make sure they are exported. Then  you should check for errors on content import.  By adding a File/Image in Plone manually you will see if there is a general issue with the instance blobstorage. 

Regards,

Volodymyr. 

On Mon, Jan 31, 2011 at 4:27 PM, Alberto Lopes <alberto-VDLNLiH7NE3QT0dZR+AlfA@public.gmane.org> wrote:

Hmmm... I just checked and (of course) the same problem happens with Files too.

On Mon, Jan 31, 2011 at 11:13 AM, Alberto Lopes <alberto-VDLNLiH7NE3QT0dZR+AlfA@public.gmane.org> wrote:

Dear Volodymyr,

About my transmogrifying migration from Plone 3->4, I just noticed another problem: none of my Images were correctly migrated. Whenever navigating to their view pages, a "POSKeyError: 'No blob file'" is generated (pastie here: http://www.pastie.org/1514570). 

Is there any option I should set for the Images migrate from filestorage to blobstorage?

Alberto

On Thu, Jan 27, 2011 at 2:11 AM, Alberto Lopes <alberto-VDLNLiH7NE3QT0dZR+AlfA@public.gmane.org> wrote:

Hi Volodymyr!

Thank you very much for your support on this migration. As I said, everything went okay. The quintagroup.transmogrifier product is excellent, I hope it (or an evolution of it) gets merged into core Plone. My main need was for something different from ZMI import/export is because ages ago I installed the contentratings product and then uninstalled it, but because it added some non-standard attributes or fields to the content objects, and because I never could get the time to write a proper script for dealing with it (it was a personal project, after all), I could never drop it from Zope. A "cleansening", standard import/export was what I needed.

One last note: the "BigFolder" I was talking about is the ATBTreeFolder, which, by default, Plone 3 created for the Members, news and events folders, if I am not mistaken.

Alberto

On Wed, Jan 26, 2011 at 4:16 PM, Volodymyr Cherepanyak <chervol-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:

Hi Alberto,

Thank you for the details. I commented inline.

On Wed, Jan 26, 2011 at 5:01 PM, Alberto Lopes <alberto-VDLNLiH7NE3QT0dZR+AlfA@public.gmane.org> wrote:

Hey, Volodymyr,

Thanks for your help so far. Indeed, the two suggestions (changing sitewalker for catalogsource and excluding the locallyAllowedTypes and so allowed me to import almost everything from my old site.

As of now, my problem is pratically solved - I can work manually from now on - but I'd like to share one or two points with you:

1 - I don't use too many collections, but for the few I have I could see that none of the criteria got imported to the new site. I will recreate them by hand. Do you think this has something to do with excluding the "locallyAllowedTypes"?

Collection criterias are not cataloged so they can be found only with 'sitewalker' pipeline.

 

2 - Mostly the workflow data was correct (I use the simple_publication_workflow), but I could see that the collection that is the default view for the Plone Site object was retracted to private state.

When you do single step Export/Import of content does not touch workflow tool. Plone4 has own settings in workflow tool that can be different from your cusom setup. So before immporting content I would advise checking if you have the workflows setup in the destination instance. 

 

3 - The BigFolder (/Members) was imported but cannot be displayed anymore (it was recreated in the destination site as a ATFolder). Traceback is here http://pastie.org/1499286. Error message is "AttributeError: 'NoneType' object has no attribute 'queryMethodID' ". This was perfectly worked-around creating a new folder and copying the contents from the broken BigFolder to the new Folder.

 

Have not spoted such issue so far. 

If 'BigFolder' is your custom content type. You can use 'substitution' pipeline to convert your BigFolders  into ATFolders

[substitution]

blueprint = quintagroup.transmogrifier.substitution

key = _type

BigFolder = ATFolder

4 - As for Mikko's tutorial, I couldn't give a path parameter to the catalogsource. If I try to use the "path = query= /Plone/sao-francisco" option as he did I get an error complaining that there are two values for the path keyword (pastie here: http://pastie.org/1499330).

Sorry, I forgot to mention that, Mikko fixed and updated some code in 'cataloguesource' recently so you need  the http://svn.plone.org/svn/collective/quintagroup.transmogrifier/trunk/  to have the path query working. 

Volodymyr.

 

Thank you very much for your help.

Alberto

On Wed, Jan 26, 2011 at 8:38 AM, Volodymyr Cherepanyak <chervol-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:

Hi Alberto,

Mikko done great job describing his experience with quitnagroup.transmogrifier here: http://blog.mfabrik.com/2010/12/27/importing-and-exporting-plone-folders-using-quintagroup-transmogrify/

The first error is connected to "Type restrictions" in folders they are not correctly imported due to some changes in that area in Plone4.  If you do not have the restriction rules manually setup on your source site then you can skip exporting them with:

[marshaller] 

blueprint = quintagroup.transmogrifier.marshaller 

exclude = 

     immediatelyAddableTypes 

     locallyAllowedTypes

I  avoided the UID error by exchanging the "sitewalker" pipeline with "catalogsource".

Regards,

Volodymyr.

2011/1/25 Alberto Lopes <alberto-VDLNLiH7NE3QT0dZR+AlfA@public.gmane.org>

Hi, Volodymyr,

I think you (and maybe someone else on the list, too) might be interested in my advances in using quintagroup.transmogrifier for a Plone 3->4 migration.

I could confirm that the problem before was indeed RAM shortage. My VPS is limited to 512MB, so I downloaded the entire Plone folder to my home computer, and then I tried to export again, this time watching both Ubuntu's memory monitor and the console (running Zope in foreground). While portal_setup was logging the export of each object, I could watch the RAM usage grow up to 800MB - and then there was no hanging or problem: the archive was downloaded without problems. From this I would suggest that when you guys have the time, you should look into this RAM creep, in order to enhance the product.

So, following the procedure in http://projects.quintagroup.com/products/wiki/quintagroup.transmogrifier/plone3-4, I installed a fresh Plone 4, activated quintagroup.transmogrifier and Marshall, and then tried to import the downloaded archive. A lot of objects seemed to be imported successfully (as far as I can tell, looking at the console), but from time to time I got messages like these:

*********************************************
2011-01-25 17:31:07 INFO IMPORTING _type=Folder, _path=Members/Saulo/ops
Exception in demarshaller section:
------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/Plone-4.0.2/buildout-cache/eggs/quintagroup.transmogrifier-0.3-py2.6.egg/quintagroup/transmogrifier/marshall.py", line 109, in __iter__
    self.atxml.demarshall(obj, data)
  File "/usr/local/Plone-4.0.2/buildout-cache/eggs/Products.Marshall-2.0-py2.6.egg/Products/Marshall/handlers/atxml.py", line 347, in demarshall
    context = self.parseContext(instance, data)
  File "/usr/local/Plone-4.0.2/buildout-cache/eggs/Products.Marshall-2.0-py2.6.egg/Products/Marshall/handlers/atxml.py", line 380, in parseContext
    self.parseXml(root, context, ns_map)
  File "/usr/local/Plone-4.0.2/buildout-cache/eggs/Products.Marshall-2.0-py2.6.egg/Products/Marshall/handlers/atxml.py", line 393, in parseXml
    node.text)
  File "/usr/local/Plone-4.0.2/buildout-cache/eggs/Products.Marshall-2.0-py2.6.egg/Products/Marshall/namespaces/atns.py", line 166, in processXmlValue
    svalues = data[value]
KeyError: 'Event'
*********************************************


I got this traceback in a lot of objects, but the import continued. However, it stopped with this message:

*********************************************
2011-01-25 17:31:08 ERROR Zope.SiteErrorLog 1295983868.750.8144937296 http://localhost:8081/Plone/portal_setup/manage_importTarball
Traceback (innermost last):
  Module ZPublisher.Publish, line 127, in publish
  Module ZPublisher.mapply, line 77, in mapply
  Module ZPublisher.Publish, line 47, in call_object
  Module Products.GenericSetup.tool, line 557, in manage_importTarball
  Module Products.GenericSetup.tool, line 330, in runAllImportStepsFromProfile
  Module Products.GenericSetup.tool, line 1085, in _runImportStepsFromContext
  Module Products.GenericSetup.tool, line 999, in _doRunImportStep
   - __traceback_info__: content_quinta
  Module quintagroup.transmogrifier.exportimport, line 142, in importSiteStructure
  Module quintagroup.transmogrifier.logger, line 30, in __iter__
  Module quintagroup.transmogrifier.portlets, line 146, in __iter__
  Module quintagroup.transmogrifier.comments, line 113, in __iter__
  Module quintagroup.transmogrifier.propertymanager, line 228, in __iter__
  Module quintagroup.transmogrifier.references, line 21, in __iter__
  Module quintagroup.transmogrifier.marshall, line 92, in __iter__
  Module quintagroup.transmogrifier.binary, line 151, in __iter__
  Module quintagroup.transmogrifier.datacorrector, line 58, in __iter__
  Module quintagroup.transmogrifier.adapters.importing, line 32, in __call__
  Module quintagroup.transmogrifier.adapters.importing, line 68, in importReferences
  Module Products.Archetypes.ClassGen, line 76, in generatedMutator
  Module Products.Archetypes.Field, line 1873, in set
   - __traceback_info__: (<ATDocument at /Plone/Members/Saulo/minha-primeira-pagina.html>, '5e399a4605face39f89f6aa99673c844', ['e74cf1fbc45e9872a50af6fc747486e4', '5e399a4605face39f89f6aa99673c844', '1f99d3d0241ee6e1e73d6e487d62f7a1'], [])
  Module Products.Archetypes.ReferenceEngine, line 319, in addReference
ReferenceException: Invalid target UID
*********************************************

Do you have any idea about what I could check to overcome this situation?

Thanks.

Alberto

On Sat, Jan 22, 2011 at 11:00 AM, Alberto Lopes <alberto-VDLNLiH7NE3QT0dZR+AlfA@public.gmane.org> wrote:

---------- Forwarded message ----------
From: Alberto Lopes <alberto <at> alopes.com>
Date: Fri, Jan 21, 2011 at 7:18 PM
Subject: Re: [Plone-Users] quintagroup.transmogrifier question

To: Volodymyr Cherepanyak <chervol-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>


Hey, Volodymyr,

Thank you for your answer.

Do you think I can do something to lower the memory usage in this case (taking out some export sections, or configuring one of them)? My site is really small (it is actually a personal test site), I am interested in exporting something like 500-800 objects, and there is virtually no traffic (75 visits last month, and I visit about 2 or 3 times a day)...

My server offers 512MB RAM; before trying to transmogrify my content, I never peaked it.

On Fri, Jan 21, 2011 at 2:30 PM, Volodymyr Cherepanyak <chervol-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:

Hi Alberto,

If zope stops responding it can be connected to the way your provider servers memory limits...

Volodymyr

On Fri, Jan 21, 2011 at 4:22 PM, Alberto Lopes <alberto-VDLNLiH7NE3QT0dZR+AlfA@public.gmane.org> wrote:

Hi, Volodymyr,

Trying with Chrome at home and IE here in the office resulted in the same problem.

One thing of note: the control panel for my VPS shows that memory utilization reached its peak at the same value as the limit for the server. Do you think that an out of memory error could explain the problem? It is strange because there is no "out of memory error" or something in the log.

Regards,

Alberto

On Fri, Jan 21, 2011 at 9:40 AM, Volodymyr Cherepanyak <chervol-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:

Hi Alberto,

thats weird. I have never experienced such issue while exporting content with quintagroup.transmogrifier (even exporting thousand of objects).  You can try:

 * different browser

 * patch the portal_setup "Export" button bound method to write direclty into a file on your server.

Regards,

Volodymyr.

2011/1/21 Alberto Lopes <alberto-VDLNLiH7NE3QT0dZR+AlfA@public.gmane.org>

Dear fellas,

I am trying to do a content migration using quintagroup.transmogrifier, and I am having a problem with the export phase.

When I execute the "Content (transmogrifier)" export step in portal_setup, the browser simply hangs. I mean, the browser UI works (I am able to scroll the main frame for portal_setup, I am able to stop loading the page, etc.), but it is like it takes forever to complete the request ( really mean forever, I went to sleep and six hours late the situation was the same ). The download box for the export package never appears.

In the Zope log, I get the following, as the last message (same as in the console, when I run Zope in the foreground):

------

2011-01-20T20:52:44 INFO EXPORTING 

Pipeline processing time: 00:01:09

513 items were generated in source sections

513 went through full pipeline

  0 were discarded in some section

It just stops there. I get no other error or warning messages.

Finally, if I close the browser and everything, Zope won't respond anymore. I point the browser to the Zope address, and it just keeps "loading". The foreground Zope won't respond to Ctrl+C anymore. Even if I exit my SSH session (with Zope running in the foreground), the process keeps alive, and even with all my browser windows closed, netstat -t still shows the HTTP connection in the "CLOSE WAIT" state forever. A simple kill won't the Zope process, I need to use a kill -9 in a new ssh session.

For the record, I am using Ubuntu 9.04, and Plone 3.2.1, CMF 2.1.2, Zope (Zope 2.10.7-final, python 2.4.5, linux2), Python 2.4.5 (#1, Mar 2 2009, 22:04:45) [GCC 4.3.2], PIL 1.1.6.

Can anyone help me solving this?

Regards,

Alberto

------------------------------------------------------------------------------
Special Offer-- Download ArcSight Logger for FREE (a $49 USD value)!
Finally, a world-class log management solution at an even better price-free!
Download using promo code Free_Logger_4_Dev2Dev. Offer expires
February 28th, so secure your free ArcSight Logger TODAY!
http://p.sf.net/sfu/arcsight-sfd2d
_______________________________________________
Plone-Users mailing list
Plone-Users-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
https://lists.sourceforge.net/lists/listinfo/plone-users

--
Volodymyr Cherepanyak
e-mail: chervol <at> gmail.com
skype : callto://chervol?chat

--
Volodymyr Cherepanyak
e-mail: chervol <at> gmail.com
skype : callto://chervol?chat

------------------------------------------------------------------------------
Special Offer-- Download ArcSight Logger for FREE (a $49 USD value)!
Finally, a world-class log management solution at an even better price-free!
Download using promo code Free_Logger_4_Dev2Dev. Offer expires
February 28th, so secure your free ArcSight Logger TODAY!
http://p.sf.net/sfu/arcsight-sfd2d
_______________________________________________
Plone-Users mailing list
Plone-Users-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
https://lists.sourceforge.net/lists/listinfo/plone-users

--
Volodymyr Cherepanyak
e-mail: chervol-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org
skype : callto://chervol?chat

--
Volodymyr Cherepanyak
e-mail: chervol-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org
skype : callto://chervol?chat

--
Volodymyr Cherepanyak
e-mail: chervol-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org
skype : callto://chervol?chat