Search Results for

# V3 API Reference — Custom Column Extraction

The V3 surface is for the cases where `GetFileSystemRecords_V2` is the wrong tool: huge datasets where you can't materialise every record at once, or queries where you only need a few columns out of the rich record schema and don't want to pay for the rest.

V3 lets you:

- Pick the columns you want by name (`EntryName`, `LogicalSize`, `Modified`, …)
- Window the read with a `StartRecord` / `RequestedCount` pair, so you can page through a million-file image at constant memory.
- Get columnar output — one contiguous array per column rather than an array of structs — which is friendlier to CSV export, Arrow / Parquet pipelines, and most analytical workloads.

Exports covered on this page: `GetAvailableProperties`, `CreatePropertySpecsFromJSON`, `GetFileSystemRecordsCustom_V2`, `GetFileSystemRecordsCustom_V3`, `FreePropertySpecArray`.

- When to choose V3 over V2
- The four-step workflow
- `GetAvailableProperties`
- `CreatePropertySpecsFromJSON`
- `GetFileSystemRecordsCustom_V3`
- `GetFileSystemRecordsCustom_V2` (no windowing)
- `FreePropertySpecArray`
- End-to-end C# example
- Reading column data by type
- Common patterns

## When to choose V3 over V2

Question | If yes, use…
--- | ---
Filesystem has more than ~500K records and you need every record? | V3 with pagination
You only need a handful of columns (e.g. `EntryName`, `LogicalSize`, `Modified`)? | V3 — skip the cost of materialising the rest
You're streaming records into a CSV / Parquet / database writer? | V3 — column-major output maps directly
You need every column on a small-to-medium image? | V2 (`GetFileSystemRecords_V2`) — convenience wins
You're prototyping or exploring an unknown image? | V2 — fewer moving parts

The V2 path materialises the entire `TFileRecord` array in one allocation sized roughly `RecordCount × ~80 bytes + sum(filename lengths) + sum(children list bytes)`. For a 10 M-record image that runs into the gigabytes; on 32-bit-clean allocators it can hit the 4 GB cap and return `RESULT_BUFFER_OVERFLOW`. V3 windowing avoids that ceiling entirely.

## The four-step workflow

     1. GetAvailableProperties      → JSON list of every column the DLL knows
     2. CreatePropertySpecsFromJSON → build a TPropertySpec[] from a name list
     3. GetFileSystemRecordsCustom_V3 (loop with StartRecord += page)
                                    → DLL writes columnar data into each spec
     4. FreePropertySpecArray       → release the whole graph in one call

Step 1 is optional once you know the column names you want. Steps 2-4 are the per-job sequence.

For tiny / one-shot workloads where you don't need pagination, `GetFileSystemRecordsCustom_V2` is V3 minus the windowing — same spec shape, same cleanup. See below.

## `GetAvailableProperties`

Returns a JSON array of every property the V3 path can extract. Use this to discover available column names without hard-coding.

Pascal signature:

    function GetAvailableProperties(out Buffer: PByte;
                                    out BufferSize: uint32): int32; cdecl;

C# binding:

    [DllImport("FEX.Core.dll", CallingConvention = CallingConvention.Cdecl)]
    public static extern int GetAvailableProperties(out IntPtr buffer, out uint bufferSize);

Returns: `RESULT_OK`, `RESULT_BUFFER_OVERFLOW`.

Buffer ownership: caller frees `buffer` with `FreeAllocatedBuffer`.

JSON shape:

    [
      {"name": "EntryName",   "type": "String",   "typeKind": "tkUString"},
      {"name": "LogicalSize", "type": "Int64",    "typeKind": "tkInt64"},
      {"name": "Modified",    "type": "DateTime", "typeKind": "tkFloat"},
      {"name": "Created",     "type": "DateTime", "typeKind": "tkFloat"},
      {"name": "Status",      "type": "UInt64",   "typeKind": "tkInt64"},
      ...
    ]

The `type` column maps onto `PROPERTY_TYPE_*` constants — see the property type table in types-and-constants.md.

This is also the starting point for the JSON you'll hand to `CreatePropertySpecsFromJSON`.

## `CreatePropertySpecsFromJSON`

Parses a JSON spec into a `TPropertySpec` array suitable for V3 extraction.

Pascal signature:

    function CreatePropertySpecsFromJSON(JSONString: PUTF8Char;
                                         out PropertySpecs: Pointer;
                                         out PropertyCount: int32): int32; cdecl;

C# binding:

    [DllImport("FEX.Core.dll", CallingConvention = CallingConvention.Cdecl, CharSet = CharSet.Ansi)]
    public static extern int CreatePropertySpecsFromJSON(
        [MarshalAs(UnmanagedType.LPStr)] string jsonString,
        out IntPtr propertySpecs,
        out int    propertyCount);

Parameters:

Name | Type | Dir | Purpose
--- | --- | --- | ---
`jsonString` | UTF-8 JSON | in | Array of property specs (see below).
`propertySpecs` | `IntPtr` | out | DLL-allocated `TPropertySpec[propertyCount]`. Free with `FreePropertySpecArray`.
`propertyCount` | `int32` | out | Number of specs parsed.

Returns: `RESULT_OK`, `RESULT_INVALID_JSON`, `RESULT_OUT_OF_MEMORY`.

JSON spec shape:

    [
      {"name": "EntryName"},
      {"name": "LogicalSize"},
      {"name": "Modified", "fieldName": "lastModified"}
    ]

Each entry needs at least `name` (a property name from `GetAvailableProperties`). Optional fields:

- `fieldName` — output alias if you want to rename the column in your consumer (the DLL still uses `name` to look up the source).
- `type` — force a `PROPERTY_TYPE_*` value if you want a different representation than the auto-detected one (rarely needed).
- `transform` — reserved for future transformation hints. Leave empty.

Limits: `propertyCount` is capped at 100. Need more columns? Run multiple V3 passes against the same dataset.

## `GetFileSystemRecordsCustom_V3`

The windowed extractor. Each call writes a chunk of the dataset into the spec array and reports the dataset's full size — so the caller knows when to stop paging.

Pascal signature:

    function GetFileSystemRecordsCustom_V3(ImageID: int32;
                                           PropertySpecs: Pointer;
                                           PropertyCount: int32;
                                           StartRecord: int32;
                                           RequestedCount: int32;
                                           out TotalRecords: uint32): int32; cdecl;

C# binding:

    [DllImport("FEX.Core.dll", CallingConvention = CallingConvention.Cdecl)]
    public static extern int GetFileSystemRecordsCustom_V3(
        int     imageId,
        IntPtr  propertySpecs,
        int     propertyCount,
        int     startRecord,
        int     requestedCount,
        out uint totalRecords);

Parameters:

Name | Type | Dir | Purpose
--- | --- | --- | ---
`imageId` | `int32` | in | Image-session ID. `ReadFileSystem` must have been called first.
`propertySpecs` | `IntPtr` | in/out | Spec array from `CreatePropertySpecsFromJSON`. The DLL writes column data into each spec's `FDataPtr`.
`propertyCount` | `int32` | in | Length of the spec array (max 100).
`startRecord` | `int32` | in | Zero-based start offset in the dataset. Pass `-1` to mean `0` (legacy convenience).
`requestedCount` | `int32` | in | Number of records to extract. Pass `-1` to mean "all remaining from `startRecord`".
`totalRecords` | `uint32` | out | Full dataset count, independent of windowing — useful for paging UI ("page 4 of 73").

Returns: `RESULT_OK`, `RESULT_IMAGEIDNOTFOUND` (-5), `RESULT_INVALID_PARAMETERS` (-18) — passed when `propertyCount <= 0`, `propertyCount > 100`, or `startRecord` ≥ `totalRecords`.

Output layout:

After a successful call, each spec has been populated:

Field | Meaning
--- | ---
`FDataPtr` | Pointer to a contiguous array of `FNumberValue` values.
`FValueSize` | Element size in bytes (e.g. 8 for `Int64`, 8 for pointer-to-string).
`FNumberValue` | Number of elements written — equal to the lesser of `RequestedCount` and `TotalRecords - StartRecord`.

The columns are aligned by row: `column[0].DataPtr[0]` and `column[1].DataPtr[0]` describe the same record. Reading values out is type-specific — see Reading column data by type.

Buffer ownership: the spec array and every `FDataPtr` it points to are owned by the DLL. Free the whole graph at the end with `FreePropertySpecArray(specs, count)`. Never call `FreeAllocatedBuffer` on a column pointer.

A second call to `GetFileSystemRecordsCustom_V3` against the same spec array re-uses it: the DLL frees the previous page's columns internally before writing the new ones. Don't try to keep two pages alive simultaneously through one spec array — copy the values you need into managed types first.

## `GetFileSystemRecordsCustom_V2` (no windowing)

Same shape as V3 minus the start/requested parameters. Returns every record in one shot. Use this when you know the dataset is small enough.

Pascal signature:

    function GetFileSystemRecordsCustom_V2(ImageID: int32;
                                           PropertySpecs: Pointer;
                                           PropertyCount: int32;
                                           out RecordCount: uint32): int32; cdecl;

C# binding:

    [DllImport("FEX.Core.dll", CallingConvention = CallingConvention.Cdecl)]
    public static extern int GetFileSystemRecordsCustom_V2(
        int     imageId,
        IntPtr  propertySpecs,
        int     propertyCount,
        out uint recordCount);

Returns: identical to V3 minus `RESULT_INVALID_PARAMETERS` for the windowing-specific cases.

Buffer ownership: identical to V3 — free with `FreePropertySpecArray`.

V3 is preferred for new code; the V2 column variant exists mainly for small workloads where the extra arguments would just be noise.

## `FreePropertySpecArray`

Releases the entire spec graph: every `FDataPtr`, every string slot inside string columns, every children list inside `ARRAYUINT32` columns, and finally the spec array itself.

Pascal signature:

    function FreePropertySpecArray(PropertySpecs: Pointer;
                                   PropertyCount: int32): int32; cdecl;

C# binding:

    [DllImport("FEX.Core.dll", CallingConvention = CallingConvention.Cdecl)]
    public static extern int FreePropertySpecArray(IntPtr propertySpecs, int propertyCount);

Returns: `RESULT_OK`, `RESULT_ERROR` (only if the underlying free raised an exception).

Pair every `CreatePropertySpecsFromJSON` with exactly one `FreePropertySpecArray`, regardless of how many V3 calls happened in between.

## End-to-end C# example

The example below paginates a million-record dataset, extracting just three columns into CSV. The pattern scales to any column count and any page size.

    using System.Globalization;
    using System.Runtime.InteropServices;

    const int PageSize = 100_000;

    [StructLayout(LayoutKind.Sequential, Pack = 1, CharSet = CharSet.Ansi)]
    public struct PropertySpec
    {
        [MarshalAs(UnmanagedType.ByValTStr, SizeConst = 64)] public string PropertyName;
        [MarshalAs(UnmanagedType.ByValTStr, SizeConst = 64)] public string FieldName;
        public int    PropertyID;
        public int    PropertyType;
        [MarshalAs(UnmanagedType.ByValTStr, SizeConst = 32)] public string Transform;

        public IntPtr DataPtr;
        public int    ValueSize;
        public int    NumberValue;
    }

    public static void ExportFileNamesAndSizesToCsv(int imageId, string csvPath)
    {
        NativeMethods.ReadFileSystem(imageId, out _);

        string specJson = """
            [
              {"name": "EntryName"},
              {"name": "LogicalSize"},
              {"name": "Modified"}
            ]
            """;

        IntPtr specs = IntPtr.Zero;
        int    count = 0;
        try
        {
            int rc = NativeMethods.CreatePropertySpecsFromJSON(specJson, out specs, out count);
            if (rc != NativeMethods.RESULT_OK)
                throw new FexCoreException("CreatePropertySpecsFromJSON", rc);

            using var writer = new StreamWriter(csvPath);
            writer.WriteLine("name,logicalSize,modifiedUtc");

            int   stride       = Marshal.SizeOf<PropertySpec>();
            int   start        = 0;
            uint  totalRecords = 0;

            while (true)
            {
                rc = NativeMethods.GetFileSystemRecordsCustom_V3(
                    imageId, specs, count, start, PageSize, out totalRecords);
                if (rc != NativeMethods.RESULT_OK)
                    throw new FexCoreException("GetFileSystemRecordsCustom_V3", rc);

                // Re-read each spec to get this page's column pointers
                var specArray = new PropertySpec[count];
                IntPtr cur = specs;
                for (int i = 0; i < count; i++)
                {
                    specArray[i] = Marshal.PtrToStructure<PropertySpec>(cur);
                    cur = IntPtr.Add(cur, stride);
                }

                int rowsThisPage = specArray[0].NumberValue;
                if (rowsThisPage <= 0) break;

                // Column 0: EntryName (string column — IntPtr per element)
                // Column 1: LogicalSize (int64)
                // Column 2: Modified (int64 FILETIME)
                for (int row = 0; row < rowsThisPage; row++)
                {
                    IntPtr namePtr = Marshal.ReadIntPtr(
                        specArray[0].DataPtr, row * specArray[0].ValueSize);
                    long size      = Marshal.ReadInt64(
                        specArray[1].DataPtr, row * specArray[1].ValueSize);
                    long modFt     = Marshal.ReadInt64(
                        specArray[2].DataPtr, row * specArray[2].ValueSize);

                    string name    = namePtr != IntPtr.Zero
                        ? Marshal.PtrToStringUTF8(namePtr) ?? ""
                        : "";
                    string modIso  = modFt > 0
                        ? DateTime.FromFileTimeUtc(modFt).ToString("o", CultureInfo.InvariantCulture)
                        : "";

                    writer.WriteLine($"{Csv(name)},{size},{modIso}");
                }

                start += rowsThisPage;
                if ((uint)start >= totalRecords) break;
            }
        }
        finally
        {
            if (specs != IntPtr.Zero)
                NativeMethods.FreePropertySpecArray(specs, count);
        }
    }

    private static string Csv(string s) =>
        s.Contains(',') || s.Contains('"') || s.Contains('\n')
            ? "\"" + s.Replace("\"", "\"\"") + "\""
            : s;

Key points to notice:

1. Re-read the spec array each iteration — the DLL updates `DataPtr`, `ValueSize`, and `NumberValue` in place every call.
2. Use `Marshal.ReadIntPtr` / `ReadInt64` / `ReadInt32` to dereference column pointers; `Marshal.PtrToStructure` is overkill for primitives.
3. One cleanup, in the outer `finally`. Don't try to free anything between pages.
4. Resolve every UTF-8 string before the next page or the cleanup — the column's pointer is in the DLL-owned region.

## Reading column data by type

`FValueSize` tells you how many bytes per slot, but you still need to know the in-memory shape. The mapping below covers every type the DLL emits.

`PROPERTY_TYPE_*` | `FValueSize` (typical) | Slot contains | Read with
--- | --- | --- | ---
`STRING` (1) | 8 | `IntPtr` to UTF-8 null-terminated string | `Marshal.ReadIntPtr(col, i*size)` → `Marshal.PtrToStringUTF8`
`INTEGER` (2) / `INT32` (11) | 4 | `int32` | `Marshal.ReadInt32(col, i*size)`
`INT64` (3) | 8 | `int64` | `Marshal.ReadInt64(col, i*size)`
`BOOLEAN` (4) | 1 | `byte` (0 or 1) | `Marshal.ReadByte(col, i*size) != 0`
`DATETIME` (5) | 8 | `int64` Windows FILETIME | `Marshal.ReadInt64` → `DateTime.FromFileTimeUtc` (skip if `<= 0`)
`GUID` (6) | 16 | 16 raw bytes | read 16 bytes → `new Guid(bytes)`
`BYTES` (7) | 8 | `IntPtr` to a byte buffer (length is format-specific) | inspect via `Marshal.Copy` once you know the length convention
`UINT16` (8) / `INT16` (9) | 2 | `uint16` / `int16` | `Marshal.ReadInt16`
`UINT32` (10) | 4 | `uint32` | `Marshal.ReadInt32` then cast
`UINT64` (12) | 8 | `uint64` | `Marshal.ReadInt64` then cast
`ARRAYUINT32` (13) | 8 | `IntPtr` to a `TChildrenList` (length-prefixed) | read the first `int32` for count, then count × `int32` for IDs

Always check `FNumberValue` for the live row count of this page — it can be less than `RequestedCount` near the end of the dataset.

## Common patterns

### CSV export, three columns, paginated

See the end-to-end example above.

### Just the file count

    NativeMethods.ReadFileSystem(imageId, out _);

    IntPtr specs = IntPtr.Zero; int count = 0;
    try
    {
        NativeMethods.CreatePropertySpecsFromJSON(@"[{""name"":""EntryName""}]", out specs, out count);

        NativeMethods.GetFileSystemRecordsCustom_V3(
            imageId, specs, count,
            startRecord: 0, requestedCount: 0,    // request zero rows…
            out uint total);                       // …but the total is still reported

        Console.WriteLine($"{total:N0} records in image");
    }
    finally
    {
        if (specs != IntPtr.Zero) NativeMethods.FreePropertySpecArray(specs, count);
    }

A `RequestedCount` of zero is a valid query — the DLL fills in `TotalRecords` without populating any column. Cheap way to size up an unfamiliar image.

### Filtering on the C# side

V3 doesn't push filter predicates into the DLL. Pull the columns you need, filter in your loop:

    for (int row = 0; row < rowsThisPage; row++)
    {
        long status = (long)Marshal.ReadInt64(specArray[3].DataPtr, row * specArray[3].ValueSize);
        if ((status & (long)StatusFlags.FILESTATUS_DELETED) == 0) continue;
        // ... handle deleted record ...
    }

If you find yourself iterating millions of rows just to filter most of them out, consider whether your column set should include the filter predicate — at that point the cost is cache-friendly sequential access rather than per-record DLL calls.

### Reusing one spec array across multiple images

Spec arrays are per-process and not bound to an image — the same `(specs, count)` pair can be passed to V3 calls against any open `ImageID`. Allocate once, reuse, free when you're done with the whole batch.