Advanced enumerators

The Blaise Pascal magazine #4 is out. Inside you can find a second part of my “enumerators” series (first page).

Background file scanning with OmniThreadLibrary

Yesterday, a reader asked if I can create a demo for the background file searcher, so here it is. To get the demo source, update your SVN copy or download this file.

The app has a simple interface.

Enter the path to be searched and the file mask in the edit field and click Scan.

Program starts scanning and displays current folder and number of found files during the process. If you click the X button during the scan, it will be aborted and program will close.

When background scanning completes, list of found files is displayed in the listbox.

During the scanning, main thread is fully active. You can move the program around, resize it, minimize, maximize and so on.

Implementation

Let’s take a look at the application in design mode.

Besides the components that are visible at runtime, there is also a TOmniEventMonitor component (named OTLMonitor) and a TTimer (tmrDisplayStatus) on the form.

When the user clicks the Scan button, a background task is created.

procedure TfrmBackgroundFileSearchDemo.btnScanClick(Sender: TObject);
begin
  FFileList := TStringList.Create;
  btnScan.Enabled := false;
  tmrDisplayStatus.Enabled := true;
  FScanTask := CreateTask(ScanFolders, 'ScanFolders')
    .MonitorWith(OTLMonitor)
    .SetParameter('FolderMask', inpFolderMask.Text)
    .Run;
end;

ScanFolders is the method that will do the scanning (in a background thread). We’ll return to it later. Task will be monitored with the OTLMonitor component so that we will receive task messages. OTLMonitor will also tell us when the task will terminate. Input folder and mask is send to the task as a parameter FolderMask and task is started.

The FFileList field is a TStringList that will contain a list of all found files.

Let’s ignore the scanner details for the moment and skip to the end of the scanning process. When task has completed its job, OTLMonitor.OnTaskTerminated is called.

procedure TfrmBackgroundFileSearchDemo.OTLMonitorTaskTerminated(
  const task: IOmniTaskControl);
begin
  tmrDisplayStatus.Enabled := false;
  outScanning.Text := '';
  outFiles.Text := IntToStr(FFileList.Count);
  lbFiles.Clear;
  lbFiles.Items.AddStrings(FFileList);
  FreeAndNil(FFileList);
  FScanTask := nil;
  btnScan.Enabled := true;
end;

At that point, number of found files is copied to the outFiles edit field and complete list is assigned to the listbox. Task reference FScanTask is then cleared, which causes the task object to be destroyed and Scan button is reenabled (it was disabled during the scanning process).

But what if the user closes the program by clicking the X button while the background scanner is active? Simple, we just catch the OnFormCloseQuery event and tell the task to terminate.

procedure TfrmBackgroundFileSearchDemo.FormCloseQuery(Sender: TObject;
  var CanClose: boolean);
begin
  if assigned(FScanTask) then begin
    FScanTask.Terminate;
    FScanTask := nil;
    CanClose := true;
  end;
end;

The Terminate method will do two things – tell the task to terminate and then wait for its termination. After that, we simply have to clear the task reference and allow the program to terminate.

Scanner

Let’s move to the scanning part now. The ScanFolders method (which is the main task method, the one we passed to the CreateTask) splits the value of the FolderMask parameter into folder and mask parts and passes them to the main worker ScanFolder.

procedure ScanFolders(const task: IOmniTask);
var
  folder: string;
  mask  : string;
begin
  mask := task.ParamByName['FolderMask'];
  folder := ExtractFilePath(mask);
  Delete(mask, 1, Length(folder));
  if folder <> '' then
    folder := IncludeTrailingPathDelimiter(folder);
  ScanFolder(task, folder, mask);
end;

ScanFolder first finds all subfolders of the selected folder and calls itself recursively for each subfolder. That means that we’ll first process deepest folders and then proceed to the top of the folder tree.

Then it sends a message MSG_SCAN_FOLDER to the main thread. As a parameter of this message it sends the name of the folder being processed. There’s nothing magical about this message – it is just an arbitrary numeric constant from range 0 .. 65534 (yes, number 65535 is reserved for internal use).

const
  MSG_SCAN_FOLDER  = 1;
  MSG_FOLDER_FILES = 2;

procedure ScanFolder(const task: IOmniTask; const folder, mask: string);
var
  err        : integer;
  folderFiles: TStringList;
  S          : TSearchRec;
begin
  err := FindFirst(folder + '*.*', faDirectory, S);
  if err = 0 then try
    repeat
      if ((S.Attr and faDirectory) <> 0) and (S.Name <> '.') and (S.Name <> '..') then
        ScanFolder(task, folder + S.Name + '\', mask);
      err := FindNext(S);
    until task.Terminated or (err <> 0);
  finally FindClose(S); end;
  task.Comm.Send(MSG_SCAN_FOLDER, folder);
  folderFiles := TStringList.Create;
  try
    err := FindFirst(folder + mask, 0, S);
    if err = 0 then try
      repeat
        folderFiles.Add(folder + S.Name);
        err := FindNext(S);
      until task.Terminated or (err <> 0);
    finally FindClose(S); end;
  finally task.Comm.Send(MSG_FOLDER_FILES, folderFiles); end;
end;

ScanFolder then runs the FindFirst/FindNext/FindClose loop for the second time to search for files in the folder. [BTW, if you want to first scan folders nearer to the root, just change the two loops and scan for files first and for folders second.] Each file is added to an internal TStringList object which was created just a moment before. When folder scan is completed, this object is sent to the main thread as parameter of the MSG_FOLDER_FILES message.

This approach – sending data for one folder – is a compromise between returning the complete set (full scanned tree), which would not provide a good feedback, and returning each file as we detect it, which would unnecessarily put a high load on the system.

Both Find loops test the status of the task.Terminated function and exit immediately if it is True. That allows us to terminate the background task when the user closes the application and OnFormCloseQuery is called.

Receiving messages

That’s all that has to be done in the background task but we still have to process the messages in the main thread. For that, we write the OTLMonitor’s OnTaskMessage event.

procedure TfrmBackgroundFileSearchDemo.OTLMonitorTaskMessage(
  const task: IOmniTaskControl);
var
  folderFiles: TStringList;
  msg        : TOmniMessage;
begin
  task.Comm.Receive(msg);
  if msg.MsgID = MSG_SCAN_FOLDER then
    FWaitingMessage := msg.MsgData
  else if msg.MsgID = MSG_FOLDER_FILES then begin
    folderFiles := TStringList(msg.MsgData.AsObject);
    FFileList.AddStrings(folderFiles);
    FreeAndNil(folderFiles);
    FWaitingCount := IntToStr(FFileList.Count);
  end;
end;

If the message is MSG_SCAN_FOLDER we just copy folder name to a local field. If the message is MSG_FOLDER_FILES, we copy file names from the parameter (which is a TStringList) to the global FFileList list and destroy the parameter. We also update a local field holding the number of currently found files.

So why don’t we directly update two edit fields on the form (one with current folder and another with number of found files)? Well, the background task can send many messages in one second (when processing folders will small number of files) and there’s no point in displaying them all – the user will never see what was displayed anyway. And it would slow down the GUI because Windows controls would be updated hundreds of times per second, which is never a good idea.

Instead of that we just store the strings to be displayed in two form fields and display them from a timer which is triggered three times per second. That will not show all scanned folders and all intermediate file count results, but will still provide the user with the sufficient feedback.

procedure TfrmBackgroundFileSearchDemo.tmrDisplayStatusTimer(Sender: TObject);
begin
  if FWaitingMessage <> '' then begin
    outScanning.Text := FWaitingMessage;
    FWaitingMessage := '';
  end;
  if FWaitingCount <> '' then begin
    outFiles.Text := FWaitingCount;
    FWaitingCount := '';
  end;
end;

And that’s all. Fully functional background file scanner that could easily be repackaged into a TComponent. And it’s all yours.

OmniThreadLibrary 1.01

OmniThreadLibrary 1.01 has been released yesterday. It is available via SVN (http://omnithreadlibrary.googlecode.com/svn/tags/release-1.01) or as a ZIP archive.

Changes since version 1.0a:

• *** Breaking interface change ***
• IOmniTask.Terminated renamed to IOmniTask.Stopped.
• New IOmniTask.Terminated that check whether the task has been requested to terminate. [demo 22]
• [GJ] Redesigned stack cotainer with better lock contention.
• [GJ] Totally redesigned queue container, which is no longer based on stack and allows multiple readers.
• Full D2009 support; D2009 packages, project files and Tests project group.
• Invoke-by-name and invoke-by-address messaging implemented [http://17slon.com/blogs/gabr/2008/10/erlangenizing-omnithreadlibrary.html, http://17slon.com/blogs/gabr/2008/10/omnithreadlibrary-using-rtti-to-call.html, demos 18 and 19]
• Implemented CreateTask(reference to function (task: ITaskControl)). [D2009 only, demo 21]
• Implemented blocking wait (ReceiveWait). [demo 19]
• Added enumerator to the IOmniTaskGroup interface.
• Implemented IOmniTaskGroup.RegisterAllWithTask and .UnregisterAllFromTask.
• Added automatic comm unregistration for IOmniTaskGroup.RegisterAllCommWith.
• Implemented IOmniTaskGroup.SendToAll.
• IOmniTaskControl.Terminate now kills the task after the timeout.
• New/updated tests/demos:
• 10_Containers
• 2 -> 2, 1 -> 4 and 4 -> 4 tests for stacks and queues.
• [1, 2, 4] -> [1, 2, 4] full tests.
• Writes CSV file with cumulative test results.
• 17_MsgWait: demo for the .MsgWait decorator.
• 18_StringMsgDispatch: Invoke demo.
• 19_StringMsgBenchmark: Invoke benchmark, ReceiveWait demo.
• 20_QuickSort: Parallel quicksort demo.
• 21_Anonymous_methods: Anonymous methods demo (D2009 only).
• 22_TerminationTest: Task termination demo.
• Message ID $FFFF is now reserved for internal purposes.
• Better default queue length calculation that takes into account OtlContainers  overhead and FastMM4 granulation.
• Bug fixed: TOmniValue.Null was not really initialized to Null.
• Bug fixed: Setting timer interval resets timer countdown.
• Bug fixed: TOmniTaskControl.Schedule always scheduled task to the global thread pool.
• Current versions of 3rd part units included.

Known problems:

• Thread pool is not stable under high  load.