# V3 API Reference — Custom Column Extraction

The V3 surface is for the cases where [`GetFileSystemRecords_V2`](v2-api.md#getfilesystemrecords_v2)
is the wrong tool: huge datasets where you can't materialise every record at
once, or queries where you only need a few columns out of the rich record
schema and don't want to pay for the rest.

V3 lets you:

- **Pick the columns** you want by name (`EntryName`, `LogicalSize`, `Modified`, …)
- **Window the read** with a `StartRecord` / `RequestedCount` pair, so you can
  page through a million-file image at constant memory.
- **Get columnar output** — one contiguous array per column rather than an
  array of structs — which is friendlier to CSV export, Arrow / Parquet
  pipelines, and most analytical workloads.

Exports covered on this page: `GetAvailableProperties`,
`CreatePropertySpecsFromJSON`, `GetFileSystemRecordsCustom_V2`,
`GetFileSystemRecordsCustom_V3`, `FreePropertySpecArray`.

- [When to choose V3 over V2](#when-to-choose-v3-over-v2)
- [The four-step workflow](#the-four-step-workflow)
- [`GetAvailableProperties`](#getavailableproperties)
- [`CreatePropertySpecsFromJSON`](#createpropertyspecsfromjson)
- [`GetFileSystemRecordsCustom_V3`](#getfilesystemrecordscustom_v3)
- [`GetFileSystemRecordsCustom_V2` (no windowing)](#getfilesystemrecordscustom_v2-no-windowing)
- [`FreePropertySpecArray`](#freepropertyspecarray)
- [End-to-end C# example](#end-to-end-c-example)
- [Reading column data by type](#reading-column-data-by-type)
- [Common patterns](#common-patterns)

---

## When to choose V3 over V2

| Question | If yes, use… |
|---|---|
| Filesystem has more than ~500K records *and* you need every record? | **V3** with pagination |
| You only need a handful of columns (e.g. `EntryName`, `LogicalSize`, `Modified`)? | **V3** — skip the cost of materialising the rest |
| You're streaming records into a CSV / Parquet / database writer? | **V3** — column-major output maps directly |
| You need every column on a small-to-medium image? | **V2** (`GetFileSystemRecords_V2`) — convenience wins |
| You're prototyping or exploring an unknown image? | **V2** — fewer moving parts |

The V2 path materialises the entire `TFileRecord` array in one allocation
sized roughly `RecordCount × ~80 bytes + sum(filename lengths) + sum(children list bytes)`.
For a 10 M-record image that runs into the gigabytes; on 32-bit-clean
allocators it can hit the 4 GB cap and return `RESULT_BUFFER_OVERFLOW`.
V3 windowing avoids that ceiling entirely.

---

## The four-step workflow

```
 1. GetAvailableProperties      → JSON list of every column the DLL knows
 2. CreatePropertySpecsFromJSON → build a TPropertySpec[] from a name list
 3. GetFileSystemRecordsCustom_V3 (loop with StartRecord += page)
                                → DLL writes columnar data into each spec
 4. FreePropertySpecArray       → release the whole graph in one call
```

Step 1 is optional once you know the column names you want. Steps 2-4 are
the per-job sequence.

For tiny / one-shot workloads where you don't need pagination,
`GetFileSystemRecordsCustom_V2` is V3 minus the windowing — same spec
shape, same cleanup. See [below](#getfilesystemrecordscustom_v2-no-windowing).

---

## `GetAvailableProperties`

Returns a JSON array of every property the V3 path can extract. Use this to
discover available column names without hard-coding.

**Pascal signature:**

```pascal
function GetAvailableProperties(out Buffer: PByte;
                                out BufferSize: uint32): int32; cdecl;
```

**C# binding:**

```csharp
[DllImport("FEX.Core.dll", CallingConvention = CallingConvention.Cdecl)]
public static extern int GetAvailableProperties(out IntPtr buffer, out uint bufferSize);
```

**Returns:** `RESULT_OK`, `RESULT_BUFFER_OVERFLOW`.

**Buffer ownership:** caller frees `buffer` with `FreeAllocatedBuffer`.

**JSON shape:**

```json
[
  {"name": "EntryName",   "type": "String",   "typeKind": "tkUString"},
  {"name": "LogicalSize", "type": "Int64",    "typeKind": "tkInt64"},
  {"name": "Modified",    "type": "DateTime", "typeKind": "tkFloat"},
  {"name": "Created",     "type": "DateTime", "typeKind": "tkFloat"},
  {"name": "Status",      "type": "UInt64",   "typeKind": "tkInt64"},
  ...
]
```

The `type` column maps onto `PROPERTY_TYPE_*` constants — see the property
type table in [types-and-constants.md](types-and-constants.md#property-type-constants).

This is also the starting point for the JSON you'll hand to
`CreatePropertySpecsFromJSON`.

---

## `CreatePropertySpecsFromJSON`

Parses a JSON spec into a `TPropertySpec` array suitable for V3 extraction.

**Pascal signature:**

```pascal
function CreatePropertySpecsFromJSON(JSONString: PUTF8Char;
                                     out PropertySpecs: Pointer;
                                     out PropertyCount: int32): int32; cdecl;
```

**C# binding:**

```csharp
[DllImport("FEX.Core.dll", CallingConvention = CallingConvention.Cdecl, CharSet = CharSet.Ansi)]
public static extern int CreatePropertySpecsFromJSON(
    [MarshalAs(UnmanagedType.LPStr)] string jsonString,
    out IntPtr propertySpecs,
    out int    propertyCount);
```

**Parameters:**

| Name | Type | Dir | Purpose |
|---|---|---|---|
| `jsonString` | UTF-8 JSON | in | Array of property specs (see below). |
| `propertySpecs` | `IntPtr` | out | DLL-allocated `TPropertySpec[propertyCount]`. **Free with `FreePropertySpecArray`.** |
| `propertyCount` | `int32` | out | Number of specs parsed. |

**Returns:** `RESULT_OK`, `RESULT_INVALID_JSON`, `RESULT_OUT_OF_MEMORY`.

**JSON spec shape:**

```json
[
  {"name": "EntryName"},
  {"name": "LogicalSize"},
  {"name": "Modified", "fieldName": "lastModified"}
]
```

Each entry needs at least `name` (a property name from
`GetAvailableProperties`). Optional fields:

- `fieldName` — output alias if you want to rename the column in your
  consumer (the DLL still uses `name` to look up the source).
- `type` — force a `PROPERTY_TYPE_*` value if you want a different
  representation than the auto-detected one (rarely needed).
- `transform` — reserved for future transformation hints. Leave empty.

**Limits:** `propertyCount` is capped at 100. Need more columns? Run
multiple V3 passes against the same dataset.

---

## `GetFileSystemRecordsCustom_V3`

The windowed extractor. Each call writes a chunk of the dataset into the
spec array and reports the dataset's full size — so the caller knows when
to stop paging.

**Pascal signature:**

```pascal
function GetFileSystemRecordsCustom_V3(ImageID: int32;
                                       PropertySpecs: Pointer;
                                       PropertyCount: int32;
                                       StartRecord: int32;
                                       RequestedCount: int32;
                                       out TotalRecords: uint32): int32; cdecl;
```

**C# binding:**

```csharp
[DllImport("FEX.Core.dll", CallingConvention = CallingConvention.Cdecl)]
public static extern int GetFileSystemRecordsCustom_V3(
    int     imageId,
    IntPtr  propertySpecs,
    int     propertyCount,
    int     startRecord,
    int     requestedCount,
    out uint totalRecords);
```

**Parameters:**

| Name | Type | Dir | Purpose |
|---|---|---|---|
| `imageId` | `int32` | in | Image-session ID. `ReadFileSystem` must have been called first. |
| `propertySpecs` | `IntPtr` | in/out | Spec array from `CreatePropertySpecsFromJSON`. The DLL writes column data into each spec's `FDataPtr`. |
| `propertyCount` | `int32` | in | Length of the spec array (max 100). |
| `startRecord` | `int32` | in | Zero-based start offset in the dataset. Pass `-1` to mean `0` (legacy convenience). |
| `requestedCount` | `int32` | in | Number of records to extract. Pass `-1` to mean "all remaining from `startRecord`". |
| `totalRecords` | `uint32` | out | Full dataset count, independent of windowing — useful for paging UI ("page 4 of 73"). |

**Returns:** `RESULT_OK`, `RESULT_IMAGEIDNOTFOUND` (-5),
`RESULT_INVALID_PARAMETERS` (-18) — passed when `propertyCount <= 0`,
`propertyCount > 100`, or `startRecord` ≥ `totalRecords`.

**Output layout:**

After a successful call, each spec has been populated:

| Field | Meaning |
|---|---|
| `FDataPtr` | Pointer to a contiguous array of `FNumberValue` values. |
| `FValueSize` | Element size in bytes (e.g. 8 for `Int64`, 8 for pointer-to-string). |
| `FNumberValue` | Number of elements written — **equal to the lesser of `RequestedCount` and `TotalRecords - StartRecord`**. |

The columns are aligned by row: `column[0].DataPtr[0]` and
`column[1].DataPtr[0]` describe the same record. Reading values out is
type-specific — see [Reading column data by type](#reading-column-data-by-type).

**Buffer ownership:** the spec array and every `FDataPtr` it points to are
owned by the DLL. Free the whole graph at the end with
`FreePropertySpecArray(specs, count)`. **Never** call `FreeAllocatedBuffer`
on a column pointer.

A second call to `GetFileSystemRecordsCustom_V3` against the same spec
array re-uses it: the DLL frees the previous page's columns internally
before writing the new ones. Don't try to keep two pages alive
simultaneously through one spec array — copy the values you need into
managed types first.

---

## `GetFileSystemRecordsCustom_V2` (no windowing)

Same shape as V3 minus the start/requested parameters. Returns every
record in one shot. Use this when you know the dataset is small enough.

**Pascal signature:**

```pascal
function GetFileSystemRecordsCustom_V2(ImageID: int32;
                                       PropertySpecs: Pointer;
                                       PropertyCount: int32;
                                       out RecordCount: uint32): int32; cdecl;
```

**C# binding:**

```csharp
[DllImport("FEX.Core.dll", CallingConvention = CallingConvention.Cdecl)]
public static extern int GetFileSystemRecordsCustom_V2(
    int     imageId,
    IntPtr  propertySpecs,
    int     propertyCount,
    out uint recordCount);
```

**Returns:** identical to V3 minus `RESULT_INVALID_PARAMETERS` for the
windowing-specific cases.

**Buffer ownership:** identical to V3 — free with `FreePropertySpecArray`.

V3 is preferred for new code; the V2 column variant exists mainly for
small workloads where the extra arguments would just be noise.

---

## `FreePropertySpecArray`

Releases the entire spec graph: every `FDataPtr`, every string slot inside
string columns, every children list inside `ARRAYUINT32` columns, and
finally the spec array itself.

**Pascal signature:**

```pascal
function FreePropertySpecArray(PropertySpecs: Pointer;
                               PropertyCount: int32): int32; cdecl;
```

**C# binding:**

```csharp
[DllImport("FEX.Core.dll", CallingConvention = CallingConvention.Cdecl)]
public static extern int FreePropertySpecArray(IntPtr propertySpecs, int propertyCount);
```

**Returns:** `RESULT_OK`, `RESULT_ERROR` (only if the underlying free
raised an exception).

Pair every `CreatePropertySpecsFromJSON` with exactly one
`FreePropertySpecArray`, regardless of how many V3 calls happened in
between.

---

## End-to-end C# example

The example below paginates a million-record dataset, extracting just three
columns into CSV. The pattern scales to any column count and any page size.

```csharp
using System.Globalization;
using System.Runtime.InteropServices;

const int PageSize = 100_000;

[StructLayout(LayoutKind.Sequential, Pack = 1, CharSet = CharSet.Ansi)]
public struct PropertySpec
{
    [MarshalAs(UnmanagedType.ByValTStr, SizeConst = 64)] public string PropertyName;
    [MarshalAs(UnmanagedType.ByValTStr, SizeConst = 64)] public string FieldName;
    public int    PropertyID;
    public int    PropertyType;
    [MarshalAs(UnmanagedType.ByValTStr, SizeConst = 32)] public string Transform;

    public IntPtr DataPtr;
    public int    ValueSize;
    public int    NumberValue;
}

public static void ExportFileNamesAndSizesToCsv(int imageId, string csvPath)
{
    NativeMethods.ReadFileSystem(imageId, out _);

    string specJson = """
        [
          {"name": "EntryName"},
          {"name": "LogicalSize"},
          {"name": "Modified"}
        ]
        """;

    IntPtr specs = IntPtr.Zero;
    int    count = 0;
    try
    {
        int rc = NativeMethods.CreatePropertySpecsFromJSON(specJson, out specs, out count);
        if (rc != NativeMethods.RESULT_OK)
            throw new FexCoreException("CreatePropertySpecsFromJSON", rc);

        using var writer = new StreamWriter(csvPath);
        writer.WriteLine("name,logicalSize,modifiedUtc");

        int   stride       = Marshal.SizeOf<PropertySpec>();
        int   start        = 0;
        uint  totalRecords = 0;

        while (true)
        {
            rc = NativeMethods.GetFileSystemRecordsCustom_V3(
                imageId, specs, count, start, PageSize, out totalRecords);
            if (rc != NativeMethods.RESULT_OK)
                throw new FexCoreException("GetFileSystemRecordsCustom_V3", rc);

            // Re-read each spec to get this page's column pointers
            var specArray = new PropertySpec[count];
            IntPtr cur = specs;
            for (int i = 0; i < count; i++)
            {
                specArray[i] = Marshal.PtrToStructure<PropertySpec>(cur);
                cur = IntPtr.Add(cur, stride);
            }

            int rowsThisPage = specArray[0].NumberValue;
            if (rowsThisPage <= 0) break;

            // Column 0: EntryName (string column — IntPtr per element)
            // Column 1: LogicalSize (int64)
            // Column 2: Modified (int64 FILETIME)
            for (int row = 0; row < rowsThisPage; row++)
            {
                IntPtr namePtr = Marshal.ReadIntPtr(
                    specArray[0].DataPtr, row * specArray[0].ValueSize);
                long size      = Marshal.ReadInt64(
                    specArray[1].DataPtr, row * specArray[1].ValueSize);
                long modFt     = Marshal.ReadInt64(
                    specArray[2].DataPtr, row * specArray[2].ValueSize);

                string name    = namePtr != IntPtr.Zero
                    ? Marshal.PtrToStringUTF8(namePtr) ?? ""
                    : "";
                string modIso  = modFt > 0
                    ? DateTime.FromFileTimeUtc(modFt).ToString("o", CultureInfo.InvariantCulture)
                    : "";

                writer.WriteLine($"{Csv(name)},{size},{modIso}");
            }

            start += rowsThisPage;
            if ((uint)start >= totalRecords) break;
        }
    }
    finally
    {
        if (specs != IntPtr.Zero)
            NativeMethods.FreePropertySpecArray(specs, count);
    }
}

private static string Csv(string s) =>
    s.Contains(',') || s.Contains('"') || s.Contains('\n')
        ? "\"" + s.Replace("\"", "\"\"") + "\""
        : s;
```

Key points to notice:

1. **Re-read the spec array each iteration** — the DLL updates `DataPtr`,
   `ValueSize`, and `NumberValue` in place every call.
2. **Use `Marshal.ReadIntPtr` / `ReadInt64` / `ReadInt32`** to dereference
   column pointers; `Marshal.PtrToStructure` is overkill for primitives.
3. **One cleanup**, in the outer `finally`. Don't try to free anything
   between pages.
4. **Resolve every UTF-8 string** before the next page or the cleanup —
   the column's pointer is in the DLL-owned region.

---

## Reading column data by type

`FValueSize` tells you how many bytes per slot, but you still need to know
the in-memory shape. The mapping below covers every type the DLL emits.

| `PROPERTY_TYPE_*` | `FValueSize` (typical) | Slot contains | Read with |
|---|---:|---|---|
| `STRING` (1) | 8 | `IntPtr` to UTF-8 null-terminated string | `Marshal.ReadIntPtr(col, i*size)` → `Marshal.PtrToStringUTF8` |
| `INTEGER` (2) / `INT32` (11) | 4 | `int32` | `Marshal.ReadInt32(col, i*size)` |
| `INT64` (3) | 8 | `int64` | `Marshal.ReadInt64(col, i*size)` |
| `BOOLEAN` (4) | 1 | `byte` (0 or 1) | `Marshal.ReadByte(col, i*size) != 0` |
| `DATETIME` (5) | 8 | `int64` Windows FILETIME | `Marshal.ReadInt64` → `DateTime.FromFileTimeUtc` (skip if `<= 0`) |
| `GUID` (6) | 16 | 16 raw bytes | read 16 bytes → `new Guid(bytes)` |
| `BYTES` (7) | 8 | `IntPtr` to a byte buffer (length is format-specific) | inspect via `Marshal.Copy` once you know the length convention |
| `UINT16` (8) / `INT16` (9) | 2 | `uint16` / `int16` | `Marshal.ReadInt16` |
| `UINT32` (10) | 4 | `uint32` | `Marshal.ReadInt32` then cast |
| `UINT64` (12) | 8 | `uint64` | `Marshal.ReadInt64` then cast |
| `ARRAYUINT32` (13) | 8 | `IntPtr` to a `TChildrenList` (length-prefixed) | read the first `int32` for count, then count × `int32` for IDs |

Always check `FNumberValue` for the live row count of *this page* — it can
be less than `RequestedCount` near the end of the dataset.

---

## Common patterns

### CSV export, three columns, paginated

See the [end-to-end example](#end-to-end-c-example) above.

### Just the file count

```csharp
NativeMethods.ReadFileSystem(imageId, out _);

IntPtr specs = IntPtr.Zero; int count = 0;
try
{
    NativeMethods.CreatePropertySpecsFromJSON(@"[{""name"":""EntryName""}]", out specs, out count);

    NativeMethods.GetFileSystemRecordsCustom_V3(
        imageId, specs, count,
        startRecord: 0, requestedCount: 0,    // request zero rows…
        out uint total);                       // …but the total is still reported

    Console.WriteLine($"{total:N0} records in image");
}
finally
{
    if (specs != IntPtr.Zero) NativeMethods.FreePropertySpecArray(specs, count);
}
```

A `RequestedCount` of zero is a valid query — the DLL fills in `TotalRecords`
without populating any column. Cheap way to size up an unfamiliar image.

### Filtering on the C# side

V3 doesn't push filter predicates into the DLL. Pull the columns you need,
filter in your loop:

```csharp
for (int row = 0; row < rowsThisPage; row++)
{
    long status = (long)Marshal.ReadInt64(specArray[3].DataPtr, row * specArray[3].ValueSize);
    if ((status & (long)StatusFlags.FILESTATUS_DELETED) == 0) continue;
    // ... handle deleted record ...
}
```

If you find yourself iterating millions of rows just to filter most of
them out, consider whether your column set should include the filter
predicate — at that point the cost is cache-friendly sequential access
rather than per-record DLL calls.

### Reusing one spec array across multiple images

Spec arrays are per-process and not bound to an image — the same
`(specs, count)` pair can be passed to V3 calls against any open
`ImageID`. Allocate once, reuse, free when you're done with the whole
batch.
