10 Feb 2012 00:43
Bypass XHDR
gypsy <gypsy <at> iswest.com>
2012-02-09 23:43:28 GMT
2012-02-09 23:43:28 GMT
First, it needs to be stated emphatically that I am not a C programmer! All I can do is copy code and then hack it until it does what I want, or I admit defeat and give up. The reason for this patch is that I wanted to suck a newsgroup with over 5 million articles; it was taking about 7 hours to run XHDR. In my opinion, XHDR is a complete waste of time for a news server because it downloads the entire article and never just the headers. In addition to the wasted 7 hours, my upstream NNTP server - newsguy - places a limitation on either the number of articles or the number of bytes downloaded in a single setting, and sends "503 transfer limit, closing connection." when that limit is reached. I didn't time it, but my guess is that the actual article download ran for 10 or 11 hours before the 503 error. Thus, each attempt consumed 17 or 18 hours, of which something like 40% was totally wasted. Worse, if I allowed snfetch to automatically restart, it would begin with the original .serial number value and download the same articles again, providing a screenfull of "oops"s and not progressing at all. My connection is very fast, via a cable modem. So I hacked snfetch to skip sending XHDR and go straight to downloading articles, beginning with the article number in .serial. In order to assure that .serial is reasonably correct, the following shell script runs on a different terminal: #! /bin/bash # This is Serial.sh. It updates .serial and terminates sn when a 503 error occurs. # The last article requested is received even though there is no 220(Continue reading)
Regards,
Chris Niekel
RSS Feed