File system watcher

I have a win program that sits in the system tray, watches for new files in a folder, and then processes and appends info to them. I am using the file system watcher control in VB.net 2005. Everything works fine when you just copy a file into the folder for testing. However, when I use the program that will be creating the file, it throws an error saying that the file is in use by another process. Is there anyway I can have my app check if the file is in use, and wait until it isn't before it does it's processing Is there something else I'm missing Right now all my processing code is triggered from the file system watcher control's file created event.

Answer this question

File system watcher

BobishKindaGuy
The problem with using Sleep() is that there is no possible value that you could give to it that would be correct all the time. What if the process using the file takes longer than 500ms. If you're processing more than one file (e.g. a list of files) why should you stop processing for x milliseconds when you could continue with the next file

I would simply add changed files to a list of files that get's processed every x milliseconds, if needed. This would require adding a background thread...

Paul1975

I don't see the need to add a special string to signify a special wait of 500ms. Just process the list when it has changed, putting the thread in a wait state when the list is empty. If a file cannot be processed (it's locked), reprocess the file(s) after a timeout of arbitrary length. For example:

Public Class Form1 Inherits System.Windows.Forms.Form
' ...
    Private files As New ArrayList
    Private terminateEvent As New AutoResetEvent(False)
    Private fileAddedEvent As New AutoResetEvent(False)

    Private Sub OnFileChanged(ByVal source As Object, ByVal eventArguments As FileSystemEventArgs)
        SyncLock (files)
            files.Add(eventArguments.FullPath)
        End SyncLock
        fileAddedEvent.Set()
    End Sub

    Private Shared Sub BackgroundThread(ByVal state As Object)
        Dim this As Form1 = state
        Dim timeout As Integer = 0
        Dim events() As AutoResetEvent = New AutoResetEvent() {this.terminateEvent, this.fileAddedEvent}
        While (True)
            Dim wait As Integer
            If timeout = 0 Then
                wait = WaitHandle.WaitAny(events)
            Else
                wait = WaitHandle.WaitAny(events, timeout, False)
            End If
            Select Case wait
                Case 0 ' terminate
                    Exit While
                Case 1, WaitHandle.WaitTimeout ' file added to list or file still needs processing
                    Debug.WriteLine("Processing List")
                    ' do something with list
                    Dim index As Integer = 0
                    SyncLock (this.files)
                        Do
                            Dim stream As FileStream
                            Try
                               stream = New FileStream(this.files(index), FileMode.Open)
                                ' TODO: process file instead of this fake
                                stream.Lock(0, stream.Length)
                                ' if we got here, the entire file could be processed
                                ' this.InvokeFileChanged(this.files(index))
                                this.files.Remove(index)
                            Catch ex As FileNotFoundException
                                this.files.RemoveAt(index)
                            Catch ex As UnauthorizedAccessException
                                Debug.WriteLine("Could not process " + this.files(index) + ", deferring...")
                                timeout = 500 ' check the list every 1/2 second
                                index = index + 1 ' process the next file
                                If (index > this.files.Count) Then
                                    ' exit if no next file
                                    Exit While
                                End If
                            Catch ex As IOException
                                Debug.WriteLine("Could not process " + this.files(index) + ", deferring...")
                                timeout = 500 ' check the list every 1/2 second
                                index = index + 1 ' process the next file
                                If (index > this.files.Count) Then
                                    ' exit if no next file
                                    Exit While
                                End If
                            Finally
                                if(Not stream Is Nothing) Then
                                    stream.Close()
                                End If
                            End Try
                        Loop While (this.files.Count > 0)
                    End SyncLock
            End Select
        End While
    End Sub
' ...
End Class

I would start the thread as follows:

ThreadPool.QueueUserWorkItem(New WaitCallback(AddressOf BackgroundThread), Me)

Where "Me" is the form object.
To exit the thread:

terminateEvent.Set()

WaitAny puts the thread into a wait state. Since we're waiting for either a terminate request or a list changed notification the thread does no processing until either of those conditions are met. If any of the files in the list cannot be processed, we skip past it and set a timeout. When WaitAny is given a timeout value, it only waits for a maximum amount of time. This is better than Sleep() because we can still come out of the wait state when one of our two conditions are met; for better performance.

PhilippeMoison
Thanks for the reply. I will have to try that out and see what happens. Unfortunately I'm building the app on my personal pc at home and have to test it at work. Makes debuging an annoyingly long process.

DavWein

I think Peter's on to something. He is right, you will need to add another thread to do the processing of the files so you can go on to another file if one is still in use. One suggestion to avoid the program from constantly retrying to open a file when it is the only one left to process is to include a non-filename string, such as ::WAIT::, in the queue to signal that the processing thread should sleep for a while before trying again.

    Private Const WaitString As String = "::WAIT::"
    Private filesToProcess As New Queue(Of String)
    Public Sub scratchpad()
        Dim curFile As String
        Dim waitLoaded As Boolean

        If filesToProcess.Count > 0 Then
            SyncLock filesToProcess
                curFile = filesToProcess.Dequeue
            End SyncLock

            Try
                If curFile = WaitString Then
                    System.Threading.Thread.Sleep(500)
                Else
                    Using fs As New IO.FileStream(curFile, IO.FileMode.Open)
                        'process file here
                    End Using
                End If
            Catch ex As Exception
                'processing failed
                'insert this file to be reprocessed
                SyncLock filesToProcess
                    If Not waitLoaded Then
                        filesToProcess.Enqueue(WaitString)
                        waitLoaded = True
                    End If
                    filesToProcess.Enqueue(curFile)
                End SyncLock
            End Try
        End If

        'wait 1 second after the queue empties to check again
        System.Threading.Thread.Sleep(1000)
    End Sub

    Private Sub fsw_Created(ByVal sender As Object, ByVal e As System.IO.FileSystemEventArgs) Handles fsw.Created
        If e.ChangeType <> IO.WatcherChangeTypes.Created Then Exit Sub
        SyncLock filesToProcess
            filesToProcess.Enqueue(e.FullPath)
        End SyncLock
    End Sub

This is, of course, only one way to do something like this. You could take out the wait condition or implement it differently, but I'm not aware of any way to check the file's availability from .NET without using a Try...Catch block. The SyncLock statements are important as they allow safe access to a shared resource (filesToProcess) from multiple threads.

lex23456
I ran into the same issue, and was able to fix it by just adding a little processing delay in the event handler to wait 0.5 seconds after the event fires before trying to use the new file:

System.Threading.Thread.Sleep(500)
File system watcher

Answer this question