Natalya_Katz | 2 Jan 09:55 2005

tiff question - performance on large files


Hello,

I tried to use libtiff for win32 (I'm working on Windows 2000, VC 6). The problem is the poor performance in case of very large files (for example, appending 100 more gray images 64x64 to tiff file with 60000 images - all of them are gray 64x64 takes about 15 minutes). I tried to use TIFFWriteScanline or TIFFWriteRawStrip, I tried to open the file with "am" flag for memory map files - nothing helped.

The question is can I improve the performance significantly (I'm not talking about 10-20%)? Can the misusage of libtiff be a reason of poor performance, or oppositely it's a well-known problem on win 2000 platform?

BR,
Natalya Katz,
Senior SW Engineer,
MIPG,
Applied Materials Israel,
972-8-9488446


The content of this message is Applied Materials Confidential.  If you are not the intended recipient and have received this message in error, any use or distribution is prohibited.  Please notify me immediately by reply e-mail and delete this message from your computer system.  Thank you.
_______________________________________________
Tiff mailing list
Tiff <at> remotesensing.org
http://xserve.flids.com/mailman/listinfo/tiff
Frank Warmerdam | 3 Jan 15:25 2005
Picon

Re: tiff question - performance on large files

On Sun, 2 Jan 2005 10:55:23 +0200, Natalya_Katz <at> amat.com
<Natalya_Katz <at> amat.com> wrote:
>  
> Hello, 
>  
> I tried to use libtiff for win32 (I'm working on Windows 2000, VC 6). The
> problem is the poor performance in case of very large files (for example,
> appending 100 more gray images 64x64 to tiff file with 60000 images - all of
> them are gray 64x64 takes about 15 minutes). I tried to use
> TIFFWriteScanline or TIFFWriteRawStrip, I tried to open the file with "am"
> flag for memory map files - nothing helped. 
>  
> The question is can I improve the performance significantly (I'm not talking
> about 10-20%)? Can the misusage of libtiff be a reason of poor performance,
> or oppositely it's a well-known problem on win 2000 platform? 

Natalya, 

I suspect the problem is due to the way libtiff just "walks" the list
of images when doing various kinds of operations rather than 
maintaining a master in-memory list of all availalbe image directories.
The current approach isn't really a problem with only a few images, 
but might well result in some quadratic performance problems when
working with as many as 60000 appended images.  

If you could prepare a test mainline that internally generated 60000
images, and that demonstrated particularly bad performance then we
might be able to look into the issue but I am not really interested in
substantially changing libtiff for this use case which is very uncommon
(in my world at least).  You might find you need to do a few hacks
on libtiff or be very careful in how you use it to get decent performance
with so many appended images. 

It is also possible that if you images are very small that the problem
is just the amount of overhead in how libtiff processes image
directories.   But without a working program to check it is hard to 
know what is hitting you. 

Best regards,

--

-- 
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up   | Frank Warmerdam, warmerdam <at> pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush    | Geospatial Programmer for Rent
Gerben Vos | 3 Jan 17:31 2005

Re: regarding compressed tiff files

Meenatchi Raja Rajeshwari wrote on Thursday, 30 December, 2004 14:34:

> i want to create compressed tiff files.
> i know how to create an uncompressed tiff file.
> is that ok if i mention the compression algorithm name in
> TIFFTAG_COMPRESSION tag?

Yes, that will work. You must set the compression tag before you start
writing the image data, and libtiff will do the work for you.

> any other change i have to make in the code which is creating 
> uncompressed tiff file.

You should use TIFFWriteEncoded{Strip,Tile}, not
TIFFWriteRaw{Strip,Tile}. Works for uncompressed TIFF as well
(because encoded and raw are the same in that case).

					Gerben Vos.
Ian Ameline | 5 Jan 22:49 2005

Re: tiff question - performance on large files

Natalya,

I regularly write and read tiffs with dozens of images in them (as layers) 
averaging 2k by 2k pixels or more, and have been satisfied with the 
performance. I've also not noticed any significant performance difference 
with LibTiff comparing Win2k, WinXP, or Mac OSX. Performance also seems 
relatively constant across compilers -- I've used VC6, VC7, Intel 7.1 and 
8.0 on Intel machines --  +- 10% or so is all I see from the optimizations 
variances among the compilers, and that mainly for the LZW or deflate codecs.

I suspect the problem is that LibTiff doesn't scale well to thousands of 
images in one tiff file. What you're doing is probably an unusual and 
unexpected (on the part of the libTifff designers) use of LibTiff.

--
Regards,			
Ian Ameline,			
Software Architect,
Alias SketchBook Technical Lead.

-- Always do the right thing. This will gratify
some people and astonish the rest. -- Mark Twain.
Lee Howard | 7 Jan 21:18 2005

JPEG (T.30-E, fax) in TIFF

Hello all.

I'm trying to get HylaFAX capable of receiving JPEG-compressed fax 
images per ITU T.30 Annex E.  So far so good, except that I cannot 
properly wrap the JPEG data into TIFF.  HylaFAX uses libtiff anyway, 
and as faxes are often multipage, storing each page in a TIFF file is a 
good way to support multipage faxes.  Thus, I need to get these JPEG 
images into a TIFF wrapper.

I've sent a color fax from a Brother MFC-3100C (to HylaFAX) and had the 
image data dumped to a file (rather than to TIFFWriteRawData as it 
would usually do).  Here is the raw, unchanged result:

   http://gateway.howardsilvan.com/fax.jpg

As I suspect is common with JPEG faxes, the SOF marker at byte 570 
doesn't indicate the image length (it's set at zero).  However, the DNL 
marker at the end of the file indicates an image length of 2128 lines.  
Most image viewers, including libjpeg do not support this ("DNL not 
supported").  So, if you fix the SOF marker to include image length, 
then most image viewers should be able to handle the image.  I've done 
this.  With a hex editor I changed the image length portion of the SOF 
marker to 2128 (changing two bytes from 0x00, 0x00 to 0x08, 0x50) to 
produce this version that should show up in most image viewers...

   http://gateway.howardsilvan.com/fax-edited.jpg

Okay, so far so good.  Now we have JPEG image data that can be viewed.  
Now I need to properly wrap it into the TIFF file.  This is where I 
have problems, and I'm hoping that some kind soul here could help out.

I don't immediately know how to analyze the JPEG image to determine the 
parameters, but following T.30-E it looks like the image should be:

   8 bits/pel/component
   4:1:1 Chrominance subsampling
   CIE Standard Illuminant D50
   Default gamut range
   200 x 200 dpi

However, the full range of possible JPEG types are discussed in T.4 
Annex E, and it may not be as I expect.  What I'm having trouble 
determining, though, is what information from the JPEG markers is 
actually needed by libtiff in the TIFF tags, and which TIFF tags need 
to be used to express that information.

Would someone kindly take a look at the URL'ed JPEG file(s) and explain 
to me which TIFF tags I need to use?  I've tried what I've thought to 
be correct, but inevitably I end up getting errors with "StripOffsets" 
and other JPEG warnings when I pass the data through tiffcp.

Thanks.

Lee.
Joris | 8 Jan 03:36 2005
Picon

Re: JPEG (T.30-E, fax) in TIFF

Lee,

> I'm trying to get HylaFAX capable of receiving JPEG-compressed fax
> images per ITU T.30 Annex E.  So far so good, except that I cannot
> properly wrap the JPEG data into TIFF.  HylaFAX uses libtiff anyway,
> and as faxes are often multipage, storing each page in a TIFF file is a
> good way to support multipage faxes.  Thus, I need to get these JPEG
> images into a TIFF wrapper.

I like your reasoning. In fact, a more general JPEG to JPEG-in-TIFF converter
might even be feasable, and useful.

> Would someone kindly take a look at the URL'ed JPEG file(s)...

I don't have the tools, nor sufficient knowledge of the JPEG specification, to
look at the files in detail.

> ...and explain
> to me which TIFF tags I need to use?

I'd say you need to make sure PhotometricInterpretation is set to 10
(PHOTOMETRIC_ITULAB), and Compression is set to 7 (COMPRESSION_JPEG). You also
need to set ImageWidth and ImageLength tags, of course. You should have one
single value of StripOffsets and StripByteCounts, and rows per strip needs to
equal ImageLength. BitsPerSample needs to be 8 (8,8,8, to be more exact, but
LibTiff should take care of that for you), SamplesPerPixel needs to be 3. IIRC,
LibTiff also requires you to set the PlanarConfiguration tag to 1
(PLANARCONFIG_CONTIG), even though the spec says that's the default value and
thus the tag should not be strictly requires.

Now, for the less obvious. You might need to set the Decode tag
(http://www.awaresystems.be/imaging/tiff/tifftags/decode.html), depending on
whether the ITULAB data in the JPEG has default range or not.

You'll also need to set the YCbCrSubSampling tag
(http://www.awaresystems.be/imaging/tiff/tifftags/ycbcrsubsampling.html), since
your subsampling values are not the default. Note that the tag's name
incorrectly seems to imply that it is for YCbCr data only, but it should also
apply in your case.

That's all I can think of.

> I've tried what I've thought to
> be correct, but inevitably I end up getting errors with "StripOffsets"
> and other JPEG warnings when I pass the data through tiffcp.

Perhaps we might be able to spot a problem if you post a code snippet.
Mentioning the exact errors and warnings might also ring a bell.

Note that few TIFF readers are ITULAB aware. I don't know if tiffcp can or
cannot handle such a TIFF you're trying to build. In fact, it may be a challenge
to find a reader that you can use to crosstest your results, and some code
failing to handle your results might not be sufficient to indicate that you
haven't yet succeeded perfectly fine...

Please do keep us posted on the outcome.

Are you a HylaFAX contributor? The HylaFAX links page still links to
libtiff.org. Can you change that, or point me to someone who can? The correct
LibTiff URL is, instead, http://www.remotesensing.org/libtiff/.

Joris Van Damme
info <at> awaresystems.be
http://www.awaresystems.be/
Download your free TIFF tag viewer for windows here:
http://www.awaresystems.be/imaging/tiff/astifftagviewer.html
Lee Howard | 8 Jan 17:36 2005

Re: JPEG (T.30-E, fax) in TIFF

Ed Grissom

On 2005.01.07 18:36 Joris wrote:

> I'd say you need to make sure PhotometricInterpretation is set to 10
> (PHOTOMETRIC_ITULAB), and Compression is set to 7 (COMPRESSION_JPEG).

I had been testing with COMPRESSION_JPEG and PHOTOMETRIC_RGB, with no 
luck, as I mentioned.  Ed Grissom suggested that I try 
PHOTOMETRIC_YCBCR with YCBCRSUBSAMPLING left at the 2:2 default, and 
immediately after I tried that I was successful in being able to pass 
the data through tiffcp and tiff2ps with the exception of two warning 
messages:

fax000003467.tif: Warning, incorrect count for field "StripOffsets" (3, 
expecting 1); tag trimmed.
fax000003467.tif: Warning, incorrect count for field "StripByteCounts" 
(3, expecting 1); tag trimmed.

(Well, to be perfectly honest, I had been using libtiff-3.6.1 up until 
last night when I upgraded to 3.7.1.  The message above is from 3.7.1.  
In 3.6.1 it said "tag ignored" instead of "tag trimmed".)  However, 
viewing the JPEG-in-TIFF image there was a very noticeable blue tint, 
much like viewing the JPEG I linked before directly in Mozilla.  I'm 
not completely certain if the blue tint indicates that the fax machine 
that I'm using to send the image is broken, if the faxed JPEG uses a 
different color pallate than is being expected, or if the fax machine 
is buggy and isn't generating the proper JPEG tables.  In any case, the 
images as-is are not preferrable to usual black-and-white faxes due to 
the blue shift; so it can't be used that way.

> You also
> need to set ImageWidth and ImageLength tags, of course. You should
> have one
> single value of StripOffsets and StripByteCounts,

Which is the cause for the warning messages I noted above: the TIFFs 
I'm generating have three values in each of those tags.  But I don't 
understand why they do.  It seems to be affected by the SAMPLESPERPIXEL 
tag.  If I change the SAMPLESPERPIXEL tag to 1 I only get one single 
value of StripOffsets and StripByteCounts.  If I just change the 
SAMPLESPERPIXEL tag to 3 then I get three values in each of 
StripOffsets and StripByteCounts.  I don't know if I'm doing something 
wrong or if there is a libtiff bug in this.

> and rows per strip needs to
> equal ImageLength.

I can't use unlimited (-1)?  I'm accustomed to using unlimited rows per 
strip since I use but one strip per page with these kinds (ECM) of 
faxes.

> BitsPerSample needs to be 8 (8,8,8, to be more
> exact, but
> LibTiff should take care of that for you), SamplesPerPixel needs to 
> be 3.

I had these before, although as I mentioned SamplesPerPixel was a bit 
troublesome, as I mentioned.

> IIRC,
> LibTiff also requires you to set the PlanarConfiguration tag to 1
> (PLANARCONFIG_CONTIG), even though the spec says that's the default
> value and
> thus the tag should not be strictly requires.

It looks like libtiff sets this tag to 1 on its own.

> Now, for the less obvious. You might need to set the Decode tag
> (http://www.awaresystems.be/imaging/tiff/tifftags/decode.html),
> depending on
> whether the ITULAB data in the JPEG has default range or not.

Aha!  Well, this probably explains why my image appears blue.  How do I 
know what the correct values are for this tag?  I've tried the said 
defaults (0, 100, -21760/255, 21590/255, -19200/255, 31800/255), but 
there's really no way for me to view the image right now due to the 
issues I mention below.  Furthermore, the tag doesn't seem to be 
getting set, despite doing:

   TIFFSetField(tif, 433, 0, 100, -21760/255, 21590/255, -19200/255, 
31800/255);

> You'll also need to set the YCbCrSubSampling tag
> (http://www.awaresystems.be/imaging/tiff/tifftags/ycbcrsubsampling.html),
> since
> your subsampling values are not the default. Note that the tag's name
> incorrectly seems to imply that it is for YCbCr data only, but it
> should also
> apply in your case.

Unfortunately, the YCBCRSUBSAMPLING tag appears to be ignored by tiffcp 
when the photometric tag is not set for YCbCr.  Observe:

[root <at> gollum recvq]# tiffdump fax000003468.tif
fax000003468.tif:
Magic: 0x4949 <little-endian> Version: 0x2a
Directory 0: offset 236712 (0x39ca8) next 0 (0)
SubFileType (254) LONG (4) 1<2>
ImageWidth (256) SHORT (3) 1<1728>
ImageLength (257) SHORT (3) 1<2128>
BitsPerSample (258) SHORT (3) 3<8 8 8>
Compression (259) SHORT (3) 1<7>
Photometric (262) SHORT (3) 1<10>
FillOrder (266) SHORT (3) 1<2>
ImageDescription (270) ASCII (2) 11<3604278160\0>
Make (271) ASCII (2) 62<LT V.92 1.0 MT5634ZBA-V9 ...>
Model (272) ASCII (2) 62<LT V.92 1.0 MT5634ZBA-V9 ...>
StripOffsets (273) LONG (4) 3<118360 0 0>
Orientation (274) SHORT (3) 1<1>
SamplesPerPixel (277) SHORT (3) 1<3>
RowsPerStrip (278) LONG (4) 1<4294967295>
StripByteCounts (279) LONG (4) 3<118352 0 0>
XResolution (282) RATIONAL (5) 1<204>
YResolution (283) RATIONAL (5) 1<196>
PlanarConfig (284) SHORT (3) 1<1>
ResolutionUnit (296) SHORT (3) 1<2>
Software (305) ASCII (2) 27<HylaFAX (tm) Version 4.2 ...>
DateTime (306) ASCII (2) 20<2005:01:08 07:26:05\0>
HostComputer (316) ASCII (2) 16<gollum.x101.com\0>
YCbCrSubsampling (530) SHORT (3) 2<1 1>
[root <at> gollum recvq]# tiffcp fax000003468.tif foo.tif
fax000003468.tif: Warning, incorrect count for field "StripOffsets" (3, 
expecting 1); tag trimmed.
fax000003468.tif: Warning, incorrect count for field "StripByteCounts" 
(3, expecting 1); tag trimmed.
JPEGPreDecode: Warning, Improper JPEG sampling factors 2,2
Apparently should be 1,1..
JPEGPreDecode: Warning, Decompressor will try reading with sampling 
2,2..
JPEGLib: Warning, Application transferred too many scanlines.

> Perhaps we might be able to spot a problem if you post a code snippet.
> Mentioning the exact errors and warnings might also ring a bell.

The errors and warnings are mentioned above.  Here is a synopsis of the 
code (most of which I didn't write - most of it probably originated 
from Sam):

     TIFFSetField(tif, TIFFTAG_SUBFILETYPE,	FILETYPE_PAGE);
     TIFFSetField(tif, TIFFTAG_IMAGEWIDTH,	(uint32) 
params.pageWidth());
     TIFFSetField(tif, TIFFTAG_BITSPERSAMPLE, 8);
     TIFFSetField(tif, TIFFTAG_PHOTOMETRIC,	PHOTOMETRIC_ITULAB);
     TIFFSetField(tif, TIFFTAG_YCBCRSUBSAMPLING, 1, 1);
     TIFFSetField(tif, 433, 0, 100, -21760/255, 21590/255, -19200/255, 
31800/255); // Decode tag
     TIFFSetField(tif, TIFFTAG_ORIENTATION,	ORIENTATION_TOPLEFT);
     TIFFSetField(tif, TIFFTAG_SAMPLESPERPIXEL, 3);
     TIFFSetField(tif, TIFFTAG_ROWSPERSTRIP,	(uint32) -1);
     TIFFSetField(tif, TIFFTAG_PLANARCONFIG,	PLANARCONFIG_CONTIG);
     TIFFSetField(tif, TIFFTAG_FILLORDER,	(uint16) fillOrder);
     TIFFSetField(tif, TIFFTAG_XRESOLUTION,	(float) 
params.horizontalRes());
     TIFFSetField(tif, TIFFTAG_YRESOLUTION,	(float) 
params.verticalRes());
     TIFFSetField(tif, TIFFTAG_RESOLUTIONUNIT,	RESUNIT_INCH);
     TIFFSetField(tif, TIFFTAG_SOFTWARE,		 
HYLAFAX_VERSION);
     TIFFSetField(tif, TIFFTAG_IMAGEDESCRIPTION,	(const char*) 
id);
     char dateTime[24];
     time_t now = Sys::now();
     strftime(dateTime, sizeof (dateTime), "%Y:%m:%d %H:%M:%S", 
localtime(&now));
     TIFFSetField(tif, TIFFTAG_DATETIME,	    dateTime);
     TIFFSetField(tif, TIFFTAG_MAKE,	    (const char*) 
getManufacturer());
     TIFFSetField(tif, TIFFTAG_MODEL,	    (const char*) getModel());
     TIFFSetField(tif, TIFFTAG_HOSTCOMPUTER, (const char*) 
server.hostname);
     TIFFSetField(tif, TIFFTAG_COMPRESSION, COMPRESSION_JPEG);
/* this appears to be done in order to collect the StripOffsets value */
     u_char null[1];
     (void) TIFFWriteRawStrip(tif, 0, null, 0);
/* this is done repeatedly until all of the data is collected */
     TIFFWriteRawStrip(tif, strip, (tdata_t)buf, cc);
/* this is done to change the reliance on the DNL marker recvEOLCount 
is not a count of EOLs, in this case it is a value that we set from 
reading the DNL marker */
     uint32* stripbytecount;
     (void) TIFFGetField(tif, TIFFTAG_STRIPBYTECOUNTS, &stripbytecount);
     uint32 sbc = stripbytecount[0];
     u_char* stripdata = new u_char[sbc];
     TIFFReadRawStrip(tif, 0, stripdata, sbc);
     for (uint32 i = 0; i < sbc-6; i++) {
         if (stripdata[i] == 0xFF && stripdata[i+1] == 0xC0 && 
stripdata[i+5] == 0x00 && stripdata[i+6] == 0x00) {
             stripdata[i+5] = recvEOLCount >> 8;
             stripdata[i+6] = recvEOLCount & 0xFF;
             protoTrace("RECV: fixing zero image frame length in SOF 
marker at byte %lu to %lu", i, recvEOLCount);
         }
     }
     TIFFSetWriteOffset(tif, 0);
     TIFFWriteRawStrip(tif, 0, (tdata_t) stripdata, sbc);
/* these tags are owned by SGI specifically for HylaFAX */
     TIFFSetField(tif, TIFFTAG_FAXRECVPARAMS, (uint32) params.encode());
     TIFFSetField(tif, TIFFTAG_FAXRECVTIME, (uint32) 
server.getPageTransferTime());

Another problem that I'll mention is that TIFFTAG_FAXRECVPARAMS and 
TIFFTAG_FAXRECVTIME are not being set as expected.  They get set when 
using traditional (black-and-white) compression methods, but not with 
JPEG compression.  I don't know if this is a hard-coded limitation in 
libtiff or not, I haven't looked into it.

> Note that few TIFF readers are ITULAB aware. I don't know if tiffcp
> can or
> cannot handle such a TIFF you're trying to build.

Well, at first looks it appears that there are some hurdles, indeed, 
but maybe I'm doing something wrong.  However, if this fax machine is 
not incorrect or broken (blue-shift), then most JPEG viewers are not 
going to work with JPEG files as I've produced them before, either.

> In fact, it may be a
> challenge
> to find a reader that you can use to crosstest your results, and some
> code
> failing to handle your results might not be sufficient to indicate
> that you
> haven't yet succeeded perfectly fine...

For the most part I'm only really interested in libtiff being able to 
support these files.  I assume that if libtiff supports them that 
tiff2ps will produce a properly-colored PostScript document that can 
then be used in whatever way the user needs.  In other words, most 
people who use HylaFAX that I work with will not use the TIFF directly 
- instead, they will post-process the fax image into PDF.  As for other 
HylaFAX users that may choose to use the TIFFs directly - for example 
in a TIFF viewer - well, unless libtiff can post-process the TIFF from 
an unusable format to a usable one for them, then they'll just need to 
leave the to-be "color fax" option in HylaFAX disabled.  I don't think 
that color faxing is done very frequently, and when it is done with any 
regularity I suspect that JBIG compression is used with greater 
frequency than JPEG.  But, since libtiff doesn't support JBIG at the 
moment, I thought that I would have an easier time of getting JPEG 
support implemented, especially since I don't have a working 
JBIG-compressed fax sender always available to me.

> Please do keep us posted on the outcome.

Will do.  As I've indicated, I'm also planning on getting JBIG support 
going, via JBIG-KIT.  I received your last message on that subject, and 
I've looked into doing it, but unfortunately I don't understand libtiff 
well enough and neither do I understand JBIG-KIT well enough to be able 
to accomplish that right now.

> Are you a HylaFAX contributor?

Yes, I'm one of regulars.  The main direction in HylaFAX development 
that I try to work involves anything with fax protocol and modem 
support, although I have done some other things.  Other people tend to 
be more affluent at working with the HylaFAX job scheduler and such.

> The HylaFAX links page still links to
> libtiff.org. Can you change that, or point me to someone who can? The
> correct
> LibTiff URL is, instead, http://www.remotesensing.org/libtiff/.

Oh, and yes, I am one of the webmasters.  I've updated the link on the 
HylaFAX HOWTO, which I believe is the only relevant one.  It may take a 
while to propagate to the live-side, though, so you may not see a 
change immediately.

What happened to libtiff.org, may I ask?

Thanks.

Lee.
Joris | 9 Jan 00:03 2005
Picon

Re: JPEG (T.30-E, fax) in TIFF

Lee,

I strongly feel that you really should state the correct information in the
Compression, Photometric, SamplesPerPixel, YCbCrSubSampling, and
StripOffsets/StripByteCounts tags. If not, you might be able to build a file
that some tool might be able to handle, but that's really an anomaly, you're
building an incorrect file really, and in five years from now even that some
tool might not be able to handle anymore. Whereas, building the file correctly,
and maybe convincing Andrey to help out extending tiffcp and the like, if he
wants to, could be a much more promising road for us all.

** Compression **

> > Compression is set to 7 (COMPRESSION_JPEG)

I guess we're clear on that one

** PhotometricInterpretation **

> > PhotometricInterpretation is set to 10 (PHOTOMETRIC_ITULAB)
>
> had been testing with PHOTOMETRIC_RGB
> Ed Grissom suggested that I try PHOTOMETRIC_YCBCR

Both are incorrect. You should really use PHOTOMETRIC_ITULAB, because the
Photometric is ITULAB.

Using YCbCr might get you passed tiffcp. But besides being a mistake, it will
solve nothing, because then next your ITULAB data will be interpreted as being
YCbCr, which it is not, and for instance convertion from YCbCr to RGB will be
applied in order to display, which will yield funny stuff but not a correct
image.

> However,
> viewing the JPEG-in-TIFF image there was a very noticeable blue tint,

Yes, this is the sort of funny stuff I'd expect. The ITULAB data is interpreted
under the incorrect impression that it is YCbCr, which the viewing software
correctly gets from your incorrect Photometric setting.

> ...much like viewing the JPEG I linked before directly in Mozilla.

This is another but related issue. Very few JPEG viewers are build to deal with
ITULAB. What most do, due to historical reasons (late arrival of JFIF standard,
poor color space specification in JPEG before that time), is simply assume a
JPEG they can't really make sense of to contain YCbCr. So, for different
reasons, these JPEG viewers are making a mistake, which is the same but correct
mistake to make for the TIFF viewers if you specify Photometric YCbCr.

** YCbCrSubSampling **

> > > 4:1:1 Chrominance subsampling
> >
> > You'll also need to set the YCbCrSubSampling tag, since your subsampling
values are not the default.
>
> YCBCRSUBSAMPLING left at the 2:2 default

At one time, Frank was asked to have LibTiff deal with TIFFs that had incorrect
YCbCrSubSampling tag values. This SubSampling information is redundant, in that
it is also present inside the JPEG compressed data. But it is needed, in that it
enabled LibTiff (and any logical TIFF codec design) to set up proper decoding
structures before starting to handle the JPEG compressed data.

What Frank did, is add code to the IFD reading/initializing section that
actually decompresses the first JPEG compressed strip/tile with LibJpeg, only to
read that SubSampling information from the JPEG compressed data, and overwrite
the value specified by the tag if necessary. This is part of the IFD
reading/initialization section, a second round of LibJpeg decompression on the
same first strip/tile is used when you call the necessary Strip/Tile reading
functions.

It might seem horrible, and it is, but it works, and solves the problem of
reading TIFFs with incorrect YCbCrSubSampling information, which aren't very
uncommon.

This explains why your bad information is forgiven by a reader. It also explains
some of the warning messages, like

> JPEGPreDecode: Warning, Improper JPEG sampling factors 2,2
> Apparently should be 1,1..
> JPEGPreDecode: Warning, Decompressor will try reading with sampling
> 2,2..

...and it might also be related to...

> JPEGLib: Warning, Application transferred too many scanlines.

...but I'm not entirely sure about the last one.

You should really specify correct YCbCrSubSampling values though, really.

** RowsPerStrip **

> > and rows per strip needs to equal ImageLength.
>
> I can't use unlimited (-1)?

Yes, you can, -1 should be perfectly fine.

** StripByteCounts/StripOffsets **

> > single value of StripOffsets and StripByteCounts
>
> fax000003467.tif: Warning, incorrect count for field "StripOffsets" (3,
> expecting 1); tag trimmed.
> fax000003467.tif: Warning, incorrect count for field "StripByteCounts"
> (3, expecting 1); tag trimmed.
> ...
> It seems to be affected by the SAMPLESPERPIXEL
> tag.  If I change the SAMPLESPERPIXEL tag to 1 I only get one single
> value of StripOffsets and StripByteCounts.  If I just change the
> SAMPLESPERPIXEL tag to 3 then I get three values in each of
> StripOffsets and StripByteCounts.  I don't know if I'm doing something
> wrong or if there is a libtiff bug in this.

This may be related to the PlanarConfiguration setting, if SamplesPerPixel
influences this. Have you tried setting PlanarConfiguration, even though you've
found LibTiff to use the proper default?

Just guessing, though.

** Decode **

> > Now, for the less obvious. You might need to set the Decode tag
> > depending on whether the ITULAB data in the JPEG has default range or not.
>
> Aha!  Well, this probably explains why my image appears blue.

I wouldn't be so sure. Past, long forgotten experience with ITULAB
color-fax-for-internet-JPEGs lead me to believe that non-default values for the
Decode information are quite rare.

The cause of your image appearing blue is most probably it is interpreted as
YCbCr instead, see above.

If your JPEG uses non-default values, they should be in that JPEG file
somewhere. It's an APP marker, but I can't remember which one or how the data
was labeled and structured... It's in a spec, somewhere, though...

** More general **

> Unfortunately, the YCBCRSUBSAMPLING tag appears to be ignored by tiffcp
> when the photometric tag is not set for YCbCr.

In my opinion, you should give up on trying to build your files such that they
are handled by tiffcp, in this first stage. If this attempt involves 'forging'
the Photometric indication, it's bound to lead to trouble later in the process,
anyway, and it build you incorrect TIFF files that you cannot hope will ever get
supported any better.

Instead, try to build a correct TIFF. You seem to have been on that track
before, (almost?) successfully. Next, try extending tiffcp, possibly Andrey
might be persuaded to help you in that area. Then, all will be well, and your
files wil be correct.

If you want, I can maybe help out by double-checking a correctly build TIFF with
independent non-LibTiff software. That's only going to be an additional
indication of the TIFF being valid, not a defenite proof, and I'll need to
extend that independent software to deal with it. But at least it is an
additional indication. Let me know.

If LibTiff doesn't write TIFFTAG_FAXRECVPARAMS and TIFFTAG_FAXRECVTIME tags for
you, I believe this too, you'll probably need to ask Andrey's help.

** The code **

The repeated calling of TIFFWriteRawStrip seems very extremely weird. Could you
try changing the DNL issue correction stuff to work on your JPEG data buffer,
before calling TIFFWriteRawStrip, and next making one single TIFFWriteRawStrip
call instead of the current three calls and the TIFFSetWriteOffset call.

** The link **

> > The HylaFAX links page still links to
> > libtiff.org. Can you change that, or point me to someone who can? The
> > correct
> > LibTiff URL is, instead, http://www.remotesensing.org/libtiff/.
>
> Oh, and yes, I am one of the webmasters.  I've updated the link on the
> HylaFAX HOWTO, which I believe is the only relevant one.  It may take a
> while to propagate to the live-side, though, so you may not see a
> change immediately.

Darren Nickerson changed the libtiff.org links on the HylaFAX links page, almost
immediatelly after I posted this note. (Thanks, Darren!) Is probably why you
couldn't find them. However, it seems you found another one. Thank you!

> What happened to libtiff.org, may I ask?

Got hijacked...

Joris Van Damme
info <at> awaresystems.be
http://www.awaresystems.be/
Download your free TIFF tag viewer for windows here:
http://www.awaresystems.be/imaging/tiff/astifftagviewer.html
Lee Howard | 9 Jan 06:46 2005

Re: JPEG (T.30-E, fax) in TIFF

On 2005.01.08 15:03 Joris wrote:

> I strongly feel that you really should state the correct information

I couldn't agree with you more.  :-)

> ** StripByteCounts/StripOffsets **
> 
> > > single value of StripOffsets and StripByteCounts
> >
> > fax000003467.tif: Warning, incorrect count for field "StripOffsets"
> (3,
> > expecting 1); tag trimmed.
> > fax000003467.tif: Warning, incorrect count for field
> "StripByteCounts"
> > (3, expecting 1); tag trimmed.
> > ...
> > It seems to be affected by the SAMPLESPERPIXEL
> > tag.  If I change the SAMPLESPERPIXEL tag to 1 I only get one single
> > value of StripOffsets and StripByteCounts.  If I just change the
> > SAMPLESPERPIXEL tag to 3 then I get three values in each of
> > StripOffsets and StripByteCounts.  I don't know if I'm doing
> something
> > wrong or if there is a libtiff bug in this.
> 
> This may be related to the PlanarConfiguration setting, if
> SamplesPerPixel
> influences this. Have you tried setting PlanarConfiguration, even
> though you've
> found LibTiff to use the proper default?

I've now tried setting TIFFTAG_PLANARCONFIG as PLANARCONFIG_CONTIG 
explicitly, and unfortunately there is no improvement.  As best I can 
tell, it looks like a bug in libtiff creating three values for 
StripByteCounts and StripOffsets when it is not supposed to.  I suppose 
that the next step to take on this one is to file a bug report?

> ** More general **
> 
> > Unfortunately, the YCBCRSUBSAMPLING tag appears to be ignored by
> tiffcp
> > when the photometric tag is not set for YCbCr.
> 
> In my opinion, you should give up on trying to build your files such
> that they
> are handled by tiffcp, in this first stage. If this attempt involves
> 'forging'
> the Photometric indication, it's bound to lead to trouble later in the
> process,
> anyway, and it build you incorrect TIFF files that you cannot hope
> will ever get
> supported any better.

Okay, you've convinced me.  I'll stop trying to use tiffcp as a gauge 
for TIFF correctness... as obviously it is broken in some degrees, when 
handling this kind of TIFF.  I was trying to use tiffcp in that way, 
though, because libtiff is really the only library that HylaFAX relies 
on for TIFF processing.

> Instead, try to build a correct TIFF.

Okay, following your advice as best as I can I have come up with:

   http://gateway.howardsilvan.com/fax000003470.tif

Here's what tiffdump tells me:

fax000003470.tif:
Magic: 0x4949 <little-endian> Version: 0x2a
Directory 0: offset 266952 (0x412c8) next 0 (0)
SubFileType (254) LONG (4) 1<2>
ImageWidth (256) SHORT (3) 1<1728>
ImageLength (257) SHORT (3) 1<2128>
BitsPerSample (258) SHORT (3) 3<8 8 8>
Compression (259) SHORT (3) 1<7>
Photometric (262) SHORT (3) 1<10>
FillOrder (266) SHORT (3) 1<2>
ImageDescription (270) ASCII (2) 11<3604278160\0>
Make (271) ASCII (2) 62<LT V.92 1.0 MT5634ZBA-V9 ...>
Model (272) ASCII (2) 62<LT V.92 1.0 MT5634ZBA-V9 ...>
StripOffsets (273) LONG (4) 3<133480 0 0>
Orientation (274) SHORT (3) 1<1>
SamplesPerPixel (277) SHORT (3) 1<3>
RowsPerStrip (278) LONG (4) 1<4294967295>
StripByteCounts (279) LONG (4) 3<133472 0 0>
XResolution (282) RATIONAL (5) 1<204>
YResolution (283) RATIONAL (5) 1<196>
PlanarConfig (284) SHORT (3) 1<1>
ResolutionUnit (296) SHORT (3) 1<2>
Software (305) ASCII (2) 27<HylaFAX (tm) Version 4.2 ...>
DateTime (306) ASCII (2) 20<2005:01:08 20:58:55\0>
HostComputer (316) ASCII (2) 16<gollum.x101.com\0>
YCbCrSubsampling (530) SHORT (3) 2<1 1>

As you can see, the StripOffsets and StripByteCounts contain three 
values each rather than one.  Looks to be a bug in libtiff as I 
mentioned above.  Also, notice that the HylaFAX-specific tags (SGI) are 
not present even though they were "set" with TIFFSetField(), as I 
mention below.

> You seem to have been on that
> track
> before, (almost?) successfully. Next, try extending tiffcp, possibly
> Andrey
> might be persuaded to help you in that area. Then, all will be well,
> and your
> files wil be correct.

Looks like I'm going to need Andrey's help on getting tiffcp enhanced 
to support PHOTOMETRIC_ITULAB.  I attempted to test this TIFF file in 
Adobe Photoshop 5.5, but Photoshop was unable to open it due to a 
"colorspace" error.

> If you want, I can maybe help out by double-checking a correctly build
> TIFF with
> independent non-LibTiff software. That's only going to be an
> additional
> indication of the TIFF being valid, not a defenite proof, and I'll
> need to
> extend that independent software to deal with it. But at least it is
> an
> additional indication. Let me know.

If you can do what you can to verify the accuracy of the TIFF I linked 
above I would appreciate it.  But, at the same time, I don't want you 
to do anything that you don't want to do.  So, don't let me abuse your 
kindness.

> If LibTiff doesn't write TIFFTAG_FAXRECVPARAMS and TIFFTAG_FAXRECVTIME
> tags for
> you, I believe this too, you'll probably need to ask Andrey's help.

Yeah.  Looks like I'll need to file a bug report on this one, also.  Or 
is Andrey listening in and does he work on these kinds of things 
without a bug report?

> ** The code **
> 
> The repeated calling of TIFFWriteRawStrip seems very extremely weird.
> Could you
> try changing the DNL issue correction stuff to work on your JPEG data
> buffer,
> before calling TIFFWriteRawStrip, and next making one single
> TIFFWriteRawStrip
> call instead of the current three calls and the TIFFSetWriteOffset
> call.

Well, the situation is like this... the fax sender will deliver to 
HylaFAX the image in a byte-by-byte stream of data that is interrupted 
by fax protocol every 64KB.  Because the SOF marker comes before the 
DNL marker, and because the SOF marker does not indicate the image 
length, it is requisite that, when the DNL marker is encountered, that 
we "go back" and "fix" the SOF marker to indicate the page length.  So, 
we have to buffer all of that data *somewhere*, and because we don't 
know how much total data we'll actually receive I didn't want to get 
into any risks of buffer overflows using RAM, and I didn't want to have 
to complicate the process by opening up a temp file on disk (yet 
another file to open and unlink later), and so chose to do what I did - 
storing the data as-it-comes in the already-open TIFF and then 
overwriting it later.

Only the "overwriting it later" part is new to HylaFAX (well, and not 
even that, because we "overwrite" the TIFF data on page retransmissions 
- when HylaFAX rejects a page and requires it to be resent).  We 
constantly write single-strip TIFFs (the usual MH, MR, or MMR 
compressions) using multiple calls to TIFFWriteRawStrip, and there 
seems to be no problem with this.  All TIFF viewers that I've tested 
have been happy with the resulting files.

Lee.
Claude Houle | 12 Jan 15:37 2005

TIFF - Unknown field with tag 34665

Hi Everyone,
 
I am transforming a TIFF Picture into a JPEG Thumbnail and I get to see this warning : Unknown field with tag 34665 encountered warning.
I am using JMagick 6.0.4
I am using ImageMagick 6.1.8-3 Q16 over Windows 2000 Professionnal Edition Service Pack 4.
 
So far, what I discovered :
 
- The Picture gives the warning when using 'identify' : Unknown field with tag 34665
- The Picture gives the warning when opening with 'IMDisplay' utility.
- The Picture have some color loss when opening with IMDisplay ( Red become blue, White becomes Black, etc... )
- The Picture works fine with Adobe Photoshop 7.0 ( The Software that originally created the picture )
- The Picture works fine and is displayable with MS Photo Editor ( Small Application that is installed with MS Office 2000 )
- The Field Tag 34665 is an Adobe Specific ( According to Magick-Users Mailing List )
- The Field Tag 34665 is not present in TIFF 6.0 Specification from 1992 ( From what I understand from the Spec )
- The Field Tag 34665 is not present in TIFF 6.0 Specification Add-On from 1995 ( From what I understand from the Spec )
- The Field Tag 34665 is not present in TIFF 6.0 Specification Add-On from 2002 ( From what I understand from the Spec )
 
How can I transform this picture without the problem ? Is this a problem with libTIFF or ImageMagick ?
 
I ruled out the JMagick Problem, because I performed the problem with ImageMagick only ( Command-Line Operations ), and the problem presented itself in the same fashion.
 
Can anyone help ?
 
Thanks,
 
Claude Houle
 
_______________________________________________
JMagick mailing list
JMagick <at> yeo.id.au
http://www.yeo.id.au/mailman/listinfo/jmagick

Gmane