Viewing composite files in Total Commander

[This is a third and probably last article in the 'embedded file system' series. If you want to refresh your memory or if you're new to my blog, read previous two installments here and here. You may also want to refresh your copy of the GpStructuredStorage unit as it was updated after the last article in the series was produced.]

I'm a great fan of Total Commander. I've been using it for who-knows-how long - I started in the Windows 3.1 times, when it was still called Windows Commander. Even before, in the DOS, I was using Norton Commander. You'd never guess, huh? ;)

One of the thing I like about Total Commander is its ability to work with various packed formats as if they were a part of a file system. You just press Enter and ZIP files opens as if it were a normal folder, and you can use standard Copy, Move, Rename etc commands on files inside the archive. Great stuff.

Another thing I like about Total Commander is its extensibility. The author has documented four types of extension interfaces (to access packed files, foreign file systems, display new file formats and to extract data from files). You have to write a DLL exporting few simple (or not) functions and plunk it into the TC plugins folder. And it "just works" after that.

The third thing I like about Total Commander is the fact that header files for plugins are all available in .h and .pas format, making Delphi implementation a snap :)

File system or packed file?

For quite some time I couldn't decide whether I should implement support for GpStructureStorage composite files as a files system plugin or as a packed file (archive) plugin. At the end, two things prevailed - packed file plugins are simpler to write and they are simpler to use (in the the Total Commander).

I started by browsing the WCX packer plugins documentation (WCX is extension for packer plugin DLLs). As WCX is simple DLL with only stdcall functions, there are no problems implementing it in Delphi. Great. I also found out that I don't have to implement much as all plugin functions are access dynamically and I can just ignore the ones I don't need. Even better!

It turns out that there are three classes of functions - for reading, writing, and packing in memory (as opposed to packing directly to the file). I only had to implement the first batch of functions as I only need read access to composite files. (Yes, writing to composite files would sometime be nice, too, but I haven't had time to implement it. You are welcom to improve my packer in that direction, if you have both time and knowledge.)

The WCX documentation told me that I only have to implement six functions:

function  OpenArchive(var ArchiveData: TOpenArchiveData): THandle; stdcall;
function  ReadHeader(hArcData: THandle; var HeaderData: THeaderData): integer; stdcall;
function  ProcessFile(hArcData: THandle; Operation: integer; DestPath, DestName: PChar): integer; stdcall;
function  CloseArchive(hArcData: THandle): integer; stdcall;
procedure SetChangeVolProc(hArcData: THandle; pChangeVolProc: TChangeVolProc); stdcall;
procedure SetProcessDataProc(hArcData: THandle; pProcessDataProc: TProcessDataProc); stdcall;

I also choose to implement two additional functions:

function  GetPackerCaps: integer; stdcall;
function  CanYouHandleThisFile(FileName: PChar): boolean; stdcall;

GetPackerCaps is really simple - it informs the Total Commander of your plugin capabilities. I specified options PK_CAPS_BY_CONTENT (TC should detect composite files by content, not by extension) and PK_CAPS_SEARCHTEXT (searching inside composite files is allowed).

function GetPackerCaps: integer;
begin
  Result := PK_CAPS_BY_CONTENT + PK_CAPS_SEARCHTEXT;
end; { GetPackerCaps }

The tricky part here is that this function is called only when plugin is installed. If you later change plugin capabilities and recompile, new capabilities won't be detected! You have to remove the plugin from TC and reinstall it so that GetPackerCaps is called again.

If there was a standard extensions for GpStructuredStorage composite files, I could just register it with Total Commander and skip the CanYouHandleThisFile function. As there is no such extension, I had to write it, but it was really simple as GpStructuredStorage already contains a function to check for valid composite file:

function CanYouHandleThisFile(FileName: PChar): boolean; stdcall;
begin
  Result := CreateStructuredStorage.IsStructuredStorage(FileName);
end; { CanYouHandleThisFile }

Now I can just select any composite file and press Enter or Ctrl+PgDn to open it inside TC. Of course, I still need to implement few functions to actually read the composite file ...

Implementing read access

To allow Total Commander to list contents of a composite file, we need to implement OpenArchive, ReadHeader, ProcessFile, and CloseArchive functions. An example from the WCX plugin documentation shows how they are used in TC to access the archive contents:

OpenArchive()          with OpenMode==PK_OM_LIST
repeat
   ReadHeader()
   ProcessFile(...,PK_SKIP,...)
until error returned
CloseArchive()

OpenArchive must return unique handle to the archive - an opaque object that is passed to ReadHeader, ProcessFile, and CloseArchive so that they know what archive they are working on. In theory, we could return IGpStructuredStorage instance converted to a THandle, but there is a problem with this - TC expects the contents of an archive to be a simple list of files and we have directory structure stored inside composite files.

Somehow, we have to walk the directory structure inside the composite file and convert it into a flat list. This could, for example, be done in the OpenArchive call. Still, we have to keep this list and the IGpStructuredStorage interface as, maybe, we will have to extract data from it.

If user copies a file from an archive, TC executes similar loop to the one above:

OpenArchive()          with OpenMode==PK_OM_EXTRACT
repeat
   ReadHeader()
   if WantToExtractThisFile()
      ProcessFile(...,PK_EXTRACT,...)
   else
      ProcessFile(...,PK_SKIP,...)
until error returned
CloseArchive()

Because some archives may not have directory and direct access to files (tar comes to mind), TC reads header of each file and then either skips or extracts it.

Therefore I have created a global list containing TGpStructuredStorageManipulator objects, each holding a reference to a IGpStructuredStorage interface plus an internal state required for the file enumeration. WCX DLL functions are just mappers into global list and manipulator objects:

function OpenArchive(var ArchiveData: TOpenArchiveData): THandle;
begin
  Result := GStructuredStorageList.OpenArchive(ArchiveData.ArcName, ArchiveData.OpenResult);
  ArchiveData.CmtBuf := nil;
  ArchiveData.CmtBufSize := 0;
  ArchiveData.CmtSize := 0;
  ArchiveData.CmtState := 0;
end; { OpenArchive }

function ReadHeader(hArcData: THandle; var HeaderData: THeaderData): integer; stdcall;
var
  manip: TGpStructuredStorageManipulator;
begin
  manip := GStructuredStorageList.LocateHandle(hArcData);
  if not assigned(manip) then
    Result := E_NOT_SUPPORTED
  else if manip.GetNextHeader(HeaderData) then
    Result := 0
  else
    Result := E_END_ARCHIVE;
end; { ReadHeader }

function ProcessFile(hArcData: THandle; Operation: integer; DestPath, DestName: PChar):
  integer; stdcall;
var
  manip  : TGpStructuredStorageManipulator;
  outFile: TFileStream;
begin
  if (Operation = PK_SKIP) or (Operation = PK_TEST) then
    Result := 0
  else begin
    manip := GStructuredStorageList.LocateHandle(hArcData);
    if not assigned(manip) then
      Result := E_NOT_SUPPORTED
    else begin
      try
        outFile := TFileStream.Create(string(DestPath) + string(DestName), fmCreate);
        try
          Result := manip.ExtractCurrentTo(outFile);
        finally FreeAndNil(outFile); end;
      except
        on E: EFCreateError do
          Result := E_ECREATE;
      end;
    end;
  end;
end; { ProcessFile } 

function CloseArchive(hArcData: THandle): integer; stdcall;
begin
  Result := GStructuredStorageList.CloseArchive(hArcData);
end; { CloseArchive }

If you'll be browsing the code, you'll notice that I decided against creating a long list of all files and folders inside the composite file in the OpenArchive procedure. Instead, I'm keeping internal enumerator state in the TGpStructuredStorageManipulator object and GetNextHeader method of this class uses this internal state to walk over the composite file. It seemed neater that way.

Oh, and what about SetChangeVolProc and SetProcessDataProc? They just set callback functions that we don't need. Read more on them in the WCX plugin documentation.

WRITING TO COMPOSITE FILES

Adding write and delete access to the plugin should be easy - one only has to implement PackFiles and DeleteFiles. Implementation should be quite straightforward as theay don't use the 'iterate until you find the correct file' approach. Instead of that, TC sends full file name (with folders and everything) to the procedure. But as I had no need for write access, I haven't implemented that part yet. Feel free to do it instead of me (and don't forget to update GetPackerCaps accordingly :) ).