headers
Ashley M. Kirchner | 2 May 2007 05:39
Favicon

Re: Monitoring files (cont.)

Charles Galpin wrote:
> ...it just sounds like you want to get the  
> file count, do a directory listing and keep checking until the count  
> matches (sleeping inbetween), then use lsof to make sure they are not  
> open  before processing them. If lsof reports them opened by another  
> process, it means they are still being copied, so you'd want to sleep  
> and check again (or keep processing the other ones and come back to it).
>   
    Question, would lsof tell me the file's open if the process that 
writing to it isn't on the same machine?  See, the files that I'm 
needing to check are being created on an NT box and all I have is access 
(through CIFS) to the folder where those files are being created in.  So 
how would lsof know whether those files are open or not if the process 
isn't on the same machine?  Magic?  ESP?

    I think, and I say this with a touch of salt, but I think I can 
simply count the files and go by that.  The container files are very 
small (less than 100 bytes) so it really doesn't take a whole lot of 
time to write a file.  So I think I'm safe in counting the files, and if 
I get a matching count, I'm done and can move on to the next step which 
is copying the actual image files over.

    -- A

--

-- 
H | It's not a bug - it's an undocumented feature.
  +--------------------------------------------------------------------
  Ashley M. Kirchner <mailto:ashley@...>   .   303.442.6410 x130
  IT Director / SysAdmin / Websmith             .     800.441.3873 x130
  Photo Craft Imaging                       .     3550 Arapahoe Ave. #6
(Continue reading)

Permalink | Reply | 6 comments |
headers
Roger | 2 May 2007 08:03
Picon

Re: Monitoring files (cont.)

On 5/1/07, Ashley M. Kirchner <ashley@...> wrote:
> Charles Galpin wrote:
> > ...it just sounds like you want to get the
> > file count, do a directory listing and keep checking until the count
> > matches (sleeping inbetween), then use lsof to make sure they are not
> > open  before processing them. If lsof reports them opened by another
> > process, it means they are still being copied, so you'd want to sleep
> > and check again (or keep processing the other ones and come back to it).
> >
>     Question, would lsof tell me the file's open if the process that
> writing to it isn't on the same machine?  See, the files that I'm
> needing to check are being created on an NT box and all I have is access
> (through CIFS) to the folder where those files are being created in.  So
> how would lsof know whether those files are open or not if the process
> isn't on the same machine?  Magic?  ESP?
I think the OS would know somehow.  the machine that you're checking
it on is running samba, right?  It's got to have the file open as a
process.   If not that, *something* on that machine has to have the
file open.    Makes sense when you stop to think about it.

--

-- 
Roger
~~~~///~~~~
Low cost web-hosting:
http://www.dreamhost.com/r.cgi?91357
Permalink | Reply | 4 comments |
headers
Ashley M. Kirchner | 2 May 2007 08:06
Favicon

Re: Monitoring files (cont.)

Roger wrote:
> I think the OS would know somehow.  the machine that you're checking
> it on is running samba, right?  It's got to have the file open as a
> process.   If not that, *something* on that machine has to have the
> file open.    Makes sense when you stop to think about it.
    The machine *I* am on, is a linux box.  The files I'm checking FOR, 
are sitting on an Windows NT box.  I can SEE the CIFS share (from that 
NT box) on the linux box (and thus see the files being created or 
removed.)  But how would lsof know what the NT box is doing or whether 
it's done writing to the file?

    -- A

--

-- 
H | It's not a bug - it's an undocumented feature.
  +--------------------------------------------------------------------
  Ashley M. Kirchner <mailto:ashley@...>   .   303.442.6410 x130
  IT Director / SysAdmin / Websmith             .     800.441.3873 x130
  Photo Craft Imaging                       .     3550 Arapahoe Ave. #6
  http://www.pcraft.com ..... .  .    .       Boulder, CO 80303, U.S.A.
Permalink | Reply | 3 comments |
headers
Roger | 2 May 2007 08:11
Picon

Re: Monitoring files (cont.)

On 5/1/07, Ashley M. Kirchner <ashley@...> wrote:
> Roger wrote:
> > I think the OS would know somehow.  the machine that you're checking
> > it on is running samba, right?  It's got to have the file open as a
> > process.   If not that, *something* on that machine has to have the
> > file open.    Makes sense when you stop to think about it.
>     The machine *I* am on, is a linux box.  The files I'm checking FOR,
> are sitting on an Windows NT box.  I can SEE the CIFS share (from that
> NT box) on the linux box (and thus see the files being created or
> removed.)  But how would lsof know what the NT box is doing or whether
> it's done writing to the file?

oh, nevermind.   guess you're back to ESP.   I was thinking you had
the share on a linux box with the windows box connecting to yours.
I'll go back to lurking now.  :^)

--

-- 
Roger
~~~~///~~~~
Low cost web-hosting:
http://www.dreamhost.com/r.cgi?91357
Permalink | Reply | 0 comments |
headers
Charles Galpin | 2 May 2007 14:19
Favicon

Re: Monitoring files (cont.)


On May 1, 2007, at 11:39 PM, Ashley M. Kirchner wrote:
>>
>     Question, would lsof tell me the file's open if the process that
> writing to it isn't on the same machine?  See, the files that I'm
> needing to check are being created on an NT box and all I have is  
> access
> (through CIFS) to the folder where those files are being created  
> in.  So
> how would lsof know whether those files are open or not if the process
> isn't on the same machine?  Magic?  ESP?

Yeah, didn't realize it was that. ESP works for me :)

>     I think, and I say this with a touch of salt, but I think I can
> simply count the files and go by that.  The container files are very
> small (less than 100 bytes) so it really doesn't take a whole lot of
> time to write a file.  So I think I'm safe in counting the files,  
> and if
> I get a matching count, I'm done and can move on to the next step  
> which
> is copying the actual image files over.

Throw a sleep in after the count to be safe (and make it  
configurable). Still sounds a bit risky to me.

hth
charles
Permalink | Reply | 0 comments |
headers
Jonathan Wilson | 3 May 2007 01:34

Re: Monitoring files (cont.)

>    The machine *I* am on, is a linux box.  The files I'm checking FOR, 
>are sitting on an Windows NT box.  I can SEE the CIFS share (from that 
>NT box) on the linux box (and thus see the files being created or 
>removed.)  But how would lsof know what the NT box is doing or whether 
>it's done writing to the file?
>
>    -- A

Move it to Linux, Man, move it to Linux . . .

--

-- 

----------------------
System Administrator - Cedar Creek Software - 
http://www.cedarcreeksoftware.com
http://jwadmin.blogspot.com
Permalink | Reply | 1 comment |
headers
Ashley M. Kirchner | 3 May 2007 02:14
Favicon

Re: Monitoring files (cont.)

Jonathan Wilson wrote:
> Move it to Linux, Man, move it to Linux . . .
    You go ahead and convince FujiFilm to do that, and I will promptly 
provide the hardware and OS to complete the task.

--

-- 
H | It's not a bug - it's an undocumented feature.
  +--------------------------------------------------------------------
  Ashley M. Kirchner <mailto:ashley@...>   .   303.442.6410 x130
  IT Director / SysAdmin / Websmith             .     800.441.3873 x130
  Photo Craft Imaging                       .     3550 Arapahoe Ave. #6
  http://www.pcraft.com ..... .  .    .       Boulder, CO 80303, U.S.A.
Permalink | Reply | 0 comments |
headers
Ashley M. Kirchner | 3 May 2007 21:01
Favicon

lsof monitoring


    Okay, moving on.  If I'm using 'lsof' to monitor open files (these 
are local files, not like my previous thread), I see a few problems.  
I'm hoping someone can offer suggestions...

    The scenario is this: I'm using any old FTP program on my computer 
to send up a bunch of image files to the server.  This can take a few 
minutes or a few hours.

    On the server, I want to fire off a script every so often (through 
crontask) that checks the upload folder for incoming files.  Once it 
finds a sub-folder containing data (it immediately assumed it's new 
data), it needs to make sure the files are complete and done transferring.

    This is where lsof comes into play.  So let's say I fire off lsof 
and have it check the folder containing the images to see if any are 
still open.  I see problems here:

    - What if, at the exact moment that lsof runs, my FTP client has 
JUST finished a file, but hasn't finished the entire transfer yet?  lsof 
might think there are no open files, so it's time to start processing 
what's in the folder.  This is wrong.

    - I guess I can invoke another sleep in the script and re-scan the 
same folder again prior to processing.  Okay, no problem, but...  What 
if, as obscure as it may sound, once again it hits that same exact 
timing as my FTP program, and once again doesn't find any open files?

    What's the best way to tackle this?  Check for file quantity as 
well?  If the quantity hasn't changed between the first and second
(Continue reading)

Permalink | Reply | 16 comments |
headers
Roger | 3 May 2007 21:11
Picon

Re: lsof monitoring

On 5/3/07, Ashley M. Kirchner <ashley@...> wrote:
>
>     Okay, moving on.  If I'm using 'lsof' to monitor open files (these
> are local files, not like my previous thread), I see a few problems.
> I'm hoping someone can offer suggestions...

well, what about checking a connection,  ie netstat -a -n.  In theory,
there should be no connection.  If you see 'established' 'time_wait'
or 'closed_wait', then there could be a connection. Not sure all the
possibilities, but could be that if there is a switch from one file to
another, one of the states should be there.

--

-- 
Roger
~~~~///~~~~
Low cost web-hosting:
http://www.dreamhost.com/r.cgi?91357
Permalink | Reply | 6 comments |
headers
Ashley M. Kirchner | 3 May 2007 21:22
Favicon

Re: lsof monitoring

Roger wrote:
> well, what about checking a connection,  ie netstat -a -n.  In theory,
> there should be no connection.  If you see 'established' 'time_wait'
> or 'closed_wait', then there could be a connection. Not sure all the
> possibilities, but could be that if there is a switch from one file to
> another, one of the states should be there.
>   
    How would I distinguish between multiple connections?  And which one 
belongs to which FTP folder?

--

-- 
W | It's not a bug - it's an undocumented feature.
  +--------------------------------------------------------------------
  Ashley M. Kirchner <mailto:ashley@...>   .   303.442.6410 x130
  IT Director / SysAdmin / Websmith             .     800.441.3873 x130
  Photo Craft Imaging                       .     3550 Arapahoe Ave. #6
  http://www.pcraft.com ..... .  .    .       Boulder, CO 80303, U.S.A.
Permalink | Reply | 5 comments |

Gmane