Writing single EXR file with very many channels?

_______________________________________________
Openexr-devel mailing list
Openexr-devel <at> nongnu.org
http://lists.nongnu.org/mailman/listinfo/openexr-devel

ilmbase new release

_______________________________________________
Openexr-devel mailing list
Openexr-devel <at> nongnu.org
http://lists.nongnu.org/mailman/listinfo/openexr-devel

Re: Writing single EXR file with very many channels?

Hi Kora,

the IlmImf library does not allow writing the file channel by channel;
all channels must be written at the same time.  The OpenEXR file layout
would channel-by-channel writing very inefficient.
However, you can write a few scanlines at a time or a few tiles at a
time; your frame buffer only has to be large enough to hold the portion
of the image you are writing.  (In the extreme, a single scan line or
a single tile is enough.)

Florian

Koraxen wrote:
> Hey guys,
> 
> I have a question regarding how to deal memory consumption and 
> multi-layered EXR files.
> 
> I need to write an EXR file that contains 10 different versions of a 
> same high res (5120 x 2880) image. This translates into an EXR file that 
> contains 10 layers x 4 channels = 40 channels.
> 
> What I am doing is:
> 1. Instantiate an exr framebuffer
> 2. Allocate memory for 40 high-res channels, and insert() them all into 
> framebuffer
> 3. Instantiate OutputFile object, and set the framebuffer to it
> 4. Write the OutputFile
> 
> The problem is that host system obviously runs out of memory during step #2.
> 
> What's needed here is the ability to:
> - Insert a channel to framebuffer
> - Write it to disk
> - Free memory
> - Repeat 40 times
> - End up with a *single* EXR file containing 40 channels
> 
> How can this be done?
> 
> Thanks a ton!
> - Kora
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> Openexr-devel mailing list
> Openexr-devel <at> nongnu.org
> http://lists.nongnu.org/mailman/listinfo/openexr-devel

Re: ilmbase new release

Hi John,

That is indeed the plan, I'd like to have a firmer plan for the release 
at the start of next year....
In the meantime, would you be so kind as to send me the changes and I 
will fold them in to the CVS tree and test out.

Many thanks in advance

Piotr

John Burnett wrote:
>
> Hello,
>
> Is there going to be a push for an updated ilmbase release?
>
> I made a few changes to fix build some warnings (silly things like int 
> vs size_t), and I’m curious if I should try to push those changes up 
> to the official project? If so, what are the guidelines for contributing?
>
> Thanks,
>
> John
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Openexr-devel mailing list
> Openexr-devel <at> nongnu.org
> http://lists.nongnu.org/mailman/listinfo/openexr-devel
>   

RE: ilmbase new release

Great to hear!  We are using 1.01 internally here, so the attached diff applies to that.  They're all silly
little things like doing explicit casts and using ((expr) != FALSE) to coerce expressions into bools.  Thanks!

John

-----Original Message-----
From: Piotr Stanczyk [mailto:pstanczyk <at> ilm.com] 
Sent: Wednesday, December 09, 2009 1:20 PM
To: John Burnett
Cc: openexr-devel <at> nongnu.org
Subject: Re: [Openexr-devel] ilmbase new release

Hi John,

That is indeed the plan, I'd like to have a firmer plan for the release 
at the start of next year....
In the meantime, would you be so kind as to send me the changes and I 
will fold them in to the CVS tree and test out.

Many thanks in advance

Piotr

John Burnett wrote:
>
> Hello,
>
> Is there going to be a push for an updated ilmbase release?
>
> I made a few changes to fix build some warnings (silly things like int 
> vs size_t), and I'm curious if I should try to push those changes up 
> to the official project? If so, what are the guidelines for contributing?
>
> Thanks,
>
> John
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Openexr-devel mailing list
> Openexr-devel <at> nongnu.org
> http://lists.nongnu.org/mailman/listinfo/openexr-devel
>   

Index: K:/dev/trunk/buildsystem/trunk/ThirdParty/Source/Libraries/ilmbase/Source/Imath/ImathVec.cpp
===================================================================
---
K:/dev/trunk/buildsystem/trunk/ThirdParty/Source/Libraries/ilmbase/Source/Imath/ImathVec.cpp	(revision 40239)
+++
K:/dev/trunk/buildsystem/trunk/ThirdParty/Source/Libraries/ilmbase/Source/Imath/ImathVec.cpp	(revision 40240)
@@ -107,7 +107,7 @@
 short
 Vec2<short>::length () const
 {
-    float lenF = Math<float>::sqrt (dot (*this));
+    float lenF = Math<float>::sqrt ((float)dot (*this));
     short lenS = (short) (lenF + 0.5f);
     return lenS;
 }
@@ -176,7 +176,7 @@
 int
 Vec2<int>::length () const
 {
-    float lenF = Math<float>::sqrt (dot (*this));
+    float lenF = Math<float>::sqrt ((float)dot (*this));
     int lenI = (int) (lenF + 0.5f);
     return lenI;
 }
@@ -314,7 +314,7 @@
 int
 Vec3<int>::length () const
 {
-    float lenF = Math<float>::sqrt (dot (*this));
+    float lenF = Math<float>::sqrt ((float)dot (*this));
     int lenI = (int) (lenF + 0.5f);
     return lenI;
 }
Index: K:/dev/trunk/buildsystem/trunk/ThirdParty/Source/Libraries/ilmbase/Source/Imath/ImathRandom.cpp
===================================================================
---
K:/dev/trunk/buildsystem/trunk/ThirdParty/Source/Libraries/ilmbase/Source/Imath/ImathRandom.cpp	(revision 40239)
+++
K:/dev/trunk/buildsystem/trunk/ThirdParty/Source/Libraries/ilmbase/Source/Imath/ImathRandom.cpp	(revision 40240)
@@ -91,9 +91,9 @@
     // We assume that sizeof (unsigned short) == 2.
     //
 
-    state[2] = x >> 32;
-    state[1] = x >> 16;
-    state[0] = x;
+    state[2] = unsigned short(x >> 32);
+    state[1] = unsigned short(x >> 16);
+    state[0] = unsigned short(x);
 }
 
 } // namespace
@@ -163,7 +163,7 @@
 srand48 (long int seed)
 {
     staticState[2] = seed >> 16;
-    staticState[1] = seed;
+    staticState[1] = unsigned short(seed);
     staticState[0] = 0x330e;
 }
 
Index: K:/dev/trunk/buildsystem/trunk/ThirdParty/Source/Libraries/ilmbase/Source/IlmThread/IlmThreadSemaphoreWin32.cpp
===================================================================
---
K:/dev/trunk/buildsystem/trunk/ThirdParty/Source/Libraries/ilmbase/Source/IlmThread/IlmThreadSemaphoreWin32.cpp	(revision 40239)
+++
K:/dev/trunk/buildsystem/trunk/ThirdParty/Source/Libraries/ilmbase/Source/IlmThread/IlmThreadSemaphoreWin32.cpp	(revision 40240)
@@ -95,7 +95,7 @@
 
 Semaphore::~Semaphore()
 {
-    bool ok = ::CloseHandle (_semaphore);
+    bool ok = ::CloseHandle (_semaphore) != FALSE;
     assert (ok);
 }
 
Index: K:/dev/trunk/buildsystem/trunk/ThirdParty/Source/Libraries/ilmbase/Source/IlmThread/IlmThreadPool.cpp
===================================================================
---
K:/dev/trunk/buildsystem/trunk/ThirdParty/Source/Libraries/ilmbase/Source/IlmThread/IlmThreadPool.cpp	(revision 40239)
+++
K:/dev/trunk/buildsystem/trunk/ThirdParty/Source/Libraries/ilmbase/Source/IlmThread/IlmThreadPool.cpp	(revision 40240)
@@ -247,7 +247,7 @@
     // an error like: "pure virtual method called"
     //
 
-    for (int i = 0; i < numThreads; i++)
+    for (size_t i = 0; i < numThreads; i++)
     {
 	taskSemaphore.post();
 	threadSemaphore.wait();
@@ -364,19 +364,19 @@
 
     Lock lock (_data->threadMutex);
 
-    if (count > _data->numThreads)
+    if ((size_t)count > _data->numThreads)
     {
 	//
         // Add more threads
 	//
 
-        while (_data->numThreads < count)
+        while (_data->numThreads < (size_t)count)
         {
             _data->threads.push_back (new WorkerThread (_data));
             _data->numThreads++;
         }
     }
-    else if (count < _data->numThreads)
+    else if ((size_t)count < _data->numThreads)
     {
 	//
 	// Wait until all existing threads are finished processing,
@@ -389,7 +389,7 @@
         // Add in new threads
 	//
 
-        while (_data->numThreads < count)
+        while (_data->numThreads < (size_t)count)
         {
             _data->threads.push_back (new WorkerThread (_data));
             _data->numThreads++;
Index: K:/dev/trunk/buildsystem/trunk/ThirdParty/Source/Libraries/ilmbase/Source/IlmThread/IlmThreadWin32.cpp
===================================================================
---
K:/dev/trunk/buildsystem/trunk/ThirdParty/Source/Libraries/ilmbase/Source/IlmThread/IlmThreadWin32.cpp	(revision 40239)
+++
K:/dev/trunk/buildsystem/trunk/ThirdParty/Source/Libraries/ilmbase/Source/IlmThread/IlmThreadWin32.cpp	(revision 40240)
@@ -76,7 +76,7 @@
 {
     DWORD status = ::WaitForSingleObject (_thread, INFINITE);
     assert (status ==  WAIT_OBJECT_0);
-    bool ok = ::CloseHandle (_thread);
+    bool ok = ::CloseHandle (_thread) != FALSE;
     assert (ok);
 }
 

_______________________________________________
Openexr-devel mailing list
Openexr-devel <at> nongnu.org
http://lists.nongnu.org/mailman/listinfo/openexr-devel

Re: Writing single EXR file with very many channels?

_______________________________________________
Openexr-devel mailing list
Openexr-devel <at> nongnu.org
http://lists.nongnu.org/mailman/listinfo/openexr-devel

Re: Writing single EXR file with very many channels?

_______________________________________________
Openexr-devel mailing list
Openexr-devel <at> nongnu.org
http://lists.nongnu.org/mailman/listinfo/openexr-devel

Re: PIZ-Compression / Multithreading

On Nov 24, 2009, at 2:56 AM, David wrote:

> i've got two little questions:
> I'm developping a c++-tool to convert OpenExr-images from Maya to  
> OpenExr-images which are optimized for Nuke (the tool creates a  
> data window and converts the images into scanline-mode with piz  
> compression).

Hey David, I think you'll find that to optimize for Nuke, you'll also  
want to switch to Zip compression.  The main reason is that Nuke's  
scanline renderer can go through and read just the lines it needs.   
Piz and Zip16 lump 16 lines together (right?), so Nuke has to read  
the whole chunk.

Also, since you'll tend to read the files many times more than you  
write them, Zip will decompress faster than Piz and you'll get a  
little speed bump on every read.

Finally, to take advantage of multithreading, you have to call  
writePixels() with a lot of scan lines.  For writing, I usually just  
do one big writePixels() call with all the lines together, but I find  
doing 128-line chunks is almost as fast.

Brendan

Re: PIZ-Compression / Multithreading

Brendan Bolles wrote:
> Hey David, I think you'll find that to optimize for Nuke, you'll also 
> want to switch to Zip compression.  The main reason is that Nuke's 
> scanline renderer can go through and read just the lines it needs.  Piz 
> and Zip16 lump 16 lines together (right?), so Nuke has to read the whole 
> chunk.

I don't know how Nuke works, but if it reads scan lines top-to-bottom or
bottom-to-top (as opposed to accessing the scan lines in random order),
then reading chunks of 16 lines might actually be faster than reading
individual lines.  Also, with 16-line chunks you tend to get slightly
better compression.

Re: PIZ-Compression / Multithreading

On Dec 11, 2009, at 3:31 PM, Florian Kainz wrote:

> I don't know how Nuke works, but if it reads scan lines top-to- 
> bottom or
> bottom-to-top (as opposed to accessing the scan lines in random  
> order),
> then reading chunks of 16 lines might actually be faster than reading
> individual lines.  Also, with 16-line chunks you tend to get slightly
> better compression.

If you're zoomed out in Nuke, the scanline renderer is only drawing  
every Nth scanline, so with Zip it will only decode and cache the  
lines it needs.  That's why Zip is a good choice for Nuke.  Virtually  
every other program reads the whole file every time, so Zip16  
performs a little better for them.

Brendan