Advanced

Virtual Lists (Virtualization)

The Virtualization component; props, the dataRequest contract, automatic refresh, templates, filtering, variable height, the rendering model and methods.

In lists of thousands of items, putting every item in the DOM hurts performance. MotifJS's Virtualization component shows lists whose row height is fixed or can be calculated in advance efficiently, by keeping only the visible items (and a few buffer rows on either side) in the DOM. It asks for the data page by page itself.

Basic usage

In the example, a folder's list of notes is drawn with a virtual list. The data is sliced page by page from the notes in the service, and each row is the note row component:

import { Component, Virtualization } from "@motifx/core";
import { NoteRow } from "./NoteRow";
import { FOLDERS, NotesService, type Folder, type Note } from "../services/notes";

export class NoteList extends Component<any, { folder: Folder; state: { query: string } }> {
  override view() {
    const notes = this.getService(NotesService)!;
    const { folder, state } = this.props;
    return (
      <Virtualization<Note>
        itemHeight={64}
        pageSize={50}
        dataRequest={async ({ page, pageSize }) => {
          const found = notes.list(folder, state.query);
          const start = page * pageSize;
          return {
            items: found.slice(start, start + pageSize),
            totalCount: found.length,
            hasMore: start + pageSize < found.length,
          };
        }}
        itemTemplate={(note) => <NoteRow note={note} folder={folder} />}
        emptyTemplate={() => <p class="list__empty">{FOLDERS[folder].empty}</p>}
      />
    );
  }
}
import { Component, ComponentBase, Virtualization, motifComponent } from "@motifx/core";
import { NoteRow } from "./NoteRow";
import { FOLDERS, NotesService, type Folder, type Note } from "../services/notes";

export class NoteList extends Component<any, { folder: Folder; state: { query: string } }> {
  override view() {
    const notes = this.getService(NotesService)!;
    const { folder, state } = this.props;
    return new Virtualization<Note>({
      itemHeight: 64,
      pageSize: 50,
      dataRequest: async ({ page, pageSize }) => {
        const found = notes.list(folder, state.query);
        const start = page * pageSize;
        return {
          items: found.slice(start, start + pageSize),
          totalCount: found.length,
          hasMore: start + pageSize < found.length,
        };
      },
      itemTemplate: (note) => new NoteRow({ note, folder }),
      emptyTemplate: () => motifComponent("p", (p: ComponentBase) => {
        p.class.add("list__empty");
        p.setText(FOLDERS[folder].empty);
      }),
    });
  }
}
import { FromService, Virtualization } from "@motifx/core";
import { NoteRow } from "./NoteRow";
import { FOLDERS, NotesService, type Folder, type Note } from "../services/notes";

export function NoteList(props: { folder: Folder; state: { query: string } }) {
  const notes = FromService(NotesService)!;
  const { folder, state } = props;

  return (
    <Virtualization<Note>
      itemHeight={64}
      pageSize={50}
      dataRequest={async ({ page, pageSize }) => {
        const found = notes.list(folder, state.query);
        const start = page * pageSize;
        return {
          items: found.slice(start, start + pageSize),
          totalCount: found.length,
          hasMore: start + pageSize < found.length,
        };
      }}
      itemTemplate={(note) => <NoteRow note={note} folder={folder} />}
      emptyTemplate={() => <p class="list__empty">{FOLDERS[folder].empty}</p>}
    />
  );
}
import { ComponentBase, FromService, Virtualization, motifComponent } from "@motifx/core";
import { NoteRow } from "./NoteRow";
import { FOLDERS, NotesService, type Folder, type Note } from "../services/notes";

export function NoteList(props: { folder: Folder; state: { query: string } }) {
  const notes = FromService(NotesService)!;
  const { folder, state } = props;

  return new Virtualization<Note>({
    itemHeight: 64,
    pageSize: 50,
    dataRequest: async ({ page, pageSize }) => {
      const found = notes.list(folder, state.query);
      const start = page * pageSize;
      return {
        items: found.slice(start, start + pageSize),
        totalCount: found.length,
        hasMore: start + pageSize < found.length,
      };
    },
    itemTemplate: (note) => motifComponent(NoteRow, { note, folder }),
    emptyTemplate: () => motifComponent("p", (p: ComponentBase) => {
      p.class.add("list__empty");
      p.setText(FOLDERS[folder].empty);
    }),
  });
}
import { Virtualization, motifOptions } from "@motifx/core";
import { NoteRow } from "./NoteRow";
import { FOLDERS, NotesService, type Folder, type Note } from "../services/notes";

export const NoteList = (props: { folder: Folder; state: { query: string } }) => motifOptions({
  el: null,

  view() {
    const notes = this.getService(NotesService)!;
    const { folder, state } = props;
    return (
      <Virtualization<Note>
        itemHeight={64}
        pageSize={50}
        dataRequest={async ({ page, pageSize }) => {
          const found = notes.list(folder, state.query);
          const start = page * pageSize;
          return {
            items: found.slice(start, start + pageSize),
            totalCount: found.length,
            hasMore: start + pageSize < found.length,
          };
        }}
        itemTemplate={(note) => <NoteRow note={note} folder={folder} />}
        emptyTemplate={() => <p class="list__empty">{FOLDERS[folder].empty}</p>}
      />
    );
  },
});
import { ComponentBase, Virtualization, motifComponent, motifOptions } from "@motifx/core";
import { NoteRow } from "./NoteRow";
import { FOLDERS, NotesService, type Folder, type Note } from "../services/notes";

export const NoteList = (props: { folder: Folder; state: { query: string } }) => motifOptions({
  el: null,

  view() {
    const notes = this.getService(NotesService)!;
    const { folder, state } = props;
    return new Virtualization<Note>({
      itemHeight: 64,
      pageSize: 50,
      dataRequest: async ({ page, pageSize }) => {
        const found = notes.list(folder, state.query);
        const start = page * pageSize;
        return {
          items: found.slice(start, start + pageSize),
          totalCount: found.length,
          hasMore: start + pageSize < found.length,
        };
      },
      itemTemplate: (note) => motifComponent(NoteRow, { note, folder }),
      emptyTemplate: () => motifComponent("p", (p: ComponentBase) => {
        p.class.add("list__empty");
        p.setText(FOLDERS[folder].empty);
      }),
    });
  },
});
import { Component, Virtualization } from "@motifx/core";
import { NoteRow } from "./NoteRow";
import { FOLDERS, NotesService } from "../services/notes";

export class NoteList extends Component {
  view() {
    const notes = this.getService(NotesService);
    const { folder, state } = this.props;
    return (
      <Virtualization
        itemHeight={64}
        pageSize={50}
        dataRequest={async ({ page, pageSize }) => {
          const found = notes.list(folder, state.query);
          const start = page * pageSize;
          return {
            items: found.slice(start, start + pageSize),
            totalCount: found.length,
            hasMore: start + pageSize < found.length,
          };
        }}
        itemTemplate={(note) => <NoteRow note={note} folder={folder} />}
        emptyTemplate={() => <p class="list__empty">{FOLDERS[folder].empty}</p>}
      />
    );
  }
}
import { Component, Virtualization, motifComponent } from "@motifx/core";
import { NoteRow } from "./NoteRow";
import { FOLDERS, NotesService } from "../services/notes";

export class NoteList extends Component {
  view() {
    const notes = this.getService(NotesService);
    const { folder, state } = this.props;
    return new Virtualization({
      itemHeight: 64,
      pageSize: 50,
      dataRequest: async ({ page, pageSize }) => {
        const found = notes.list(folder, state.query);
        const start = page * pageSize;
        return {
          items: found.slice(start, start + pageSize),
          totalCount: found.length,
          hasMore: start + pageSize < found.length,
        };
      },
      itemTemplate: (note) => new NoteRow({ note, folder }),
      emptyTemplate: () => motifComponent("p", (p) => {
        p.class.add("list__empty");
        p.setText(FOLDERS[folder].empty);
      }),
    });
  }
}
import { FromService, Virtualization } from "@motifx/core";
import { NoteRow } from "./NoteRow";
import { FOLDERS, NotesService } from "../services/notes";

export function NoteList(props) {
  const notes = FromService(NotesService);
  const { folder, state } = props;

  return (
    <Virtualization
      itemHeight={64}
      pageSize={50}
      dataRequest={async ({ page, pageSize }) => {
        const found = notes.list(folder, state.query);
        const start = page * pageSize;
        return {
          items: found.slice(start, start + pageSize),
          totalCount: found.length,
          hasMore: start + pageSize < found.length,
        };
      }}
      itemTemplate={(note) => <NoteRow note={note} folder={folder} />}
      emptyTemplate={() => <p class="list__empty">{FOLDERS[folder].empty}</p>}
    />
  );
}
import { FromService, Virtualization, motifComponent } from "@motifx/core";
import { NoteRow } from "./NoteRow";
import { FOLDERS, NotesService } from "../services/notes";

export function NoteList(props) {
  const notes = FromService(NotesService);
  const { folder, state } = props;

  return new Virtualization({
    itemHeight: 64,
    pageSize: 50,
    dataRequest: async ({ page, pageSize }) => {
      const found = notes.list(folder, state.query);
      const start = page * pageSize;
      return {
        items: found.slice(start, start + pageSize),
        totalCount: found.length,
        hasMore: start + pageSize < found.length,
      };
    },
    itemTemplate: (note) => motifComponent(NoteRow, { note, folder }),
    emptyTemplate: () => motifComponent("p", (p) => {
      p.class.add("list__empty");
      p.setText(FOLDERS[folder].empty);
    }),
  });
}
import { Virtualization, motifOptions } from "@motifx/core";
import { NoteRow } from "./NoteRow";
import { FOLDERS, NotesService } from "../services/notes";

export const NoteList = (props) => motifOptions({
  el: null,

  view() {
    const notes = this.getService(NotesService);
    const { folder, state } = props;
    return (
      <Virtualization
        itemHeight={64}
        pageSize={50}
        dataRequest={async ({ page, pageSize }) => {
          const found = notes.list(folder, state.query);
          const start = page * pageSize;
          return {
            items: found.slice(start, start + pageSize),
            totalCount: found.length,
            hasMore: start + pageSize < found.length,
          };
        }}
        itemTemplate={(note) => <NoteRow note={note} folder={folder} />}
        emptyTemplate={() => <p class="list__empty">{FOLDERS[folder].empty}</p>}
      />
    );
  },
});
import { Virtualization, motifComponent, motifOptions } from "@motifx/core";
import { NoteRow } from "./NoteRow";
import { FOLDERS, NotesService } from "../services/notes";

export const NoteList = (props) => motifOptions({
  el: null,

  view() {
    const notes = this.getService(NotesService);
    const { folder, state } = props;
    return new Virtualization({
      itemHeight: 64,
      pageSize: 50,
      dataRequest: async ({ page, pageSize }) => {
        const found = notes.list(folder, state.query);
        const start = page * pageSize;
        return {
          items: found.slice(start, start + pageSize),
          totalCount: found.length,
          hasMore: start + pageSize < found.length,
        };
      },
      itemTemplate: (note) => motifComponent(NoteRow, { note, folder }),
      emptyTemplate: () => motifComponent("p", (p) => {
        p.class.add("list__empty");
        p.setText(FOLDERS[folder].empty);
      }),
    });
  },
});

The folder layout places the list together with the search state with <NoteList folder={this.folder} state={this.state} />.

The folder layout places the list together with the search state with new NoteList({ folder: this.folder, state: this.state }).

The folder layout places the list together with the search state with motifComponent(NoteList, { folder: this.folder, state: this.state }).

When mainTemplate is not given, the list sets up its root div with overflow-y: auto; height: 100%; position: relative; overflow-anchor: none and scrolls inside itself, so the container that holds it must have a height, or the height is given to the list with style. dataRequest reads the search text and the notes, so the list refreshes on its own as you type into the search box (see Automatic refresh).

Props (VirtualizationProps<T>)

Prop Type Description
dataRequest (req) => Promise<VirtualizationDataResponse<T>> Required. The function that fetches a page of data; see The dataRequest contract.
itemTemplate (item: T, index: number) => any Required. Builds the row for each item. If it does not return a component, the returned value is wrapped in one. The item arrives exactly as it was given in the data (see Data is used as given).
itemHeight number \| (item: T, index: number) => number Required. The pixel height of each item. A number: all rows are the same; a function: a known height per row (see Variable height).
pageSize number How many items to ask for in each request (50 by default).
mainTemplate (content, state) => any The layout that wraps the list; see A custom container.
loadingTemplate (state) => any The view shown while the first load is in progress.
emptyTemplate () => any The view shown when there are no items left to show (the data is empty, or the filter removed all of them).
errorTemplate (error) => any The view shown in place of the rows when the first load, refresh() or an automatic refresh request fails; if it is not given, the error is reported as MJX207. An error in the next page request is always reported as MJX207 and written to getState().error; the loaded rows stay on screen, the error view is not shown, the request is made again on the next scroll, and error is cleared on a successful load.
filter (items: T[]) => T[] Applied to the loaded data; spacers and indices are calculated from the filtered length (see Filtering).
overscan number The number of extra rows drawn above and below the visible area (by default, the number of visible rows).
cacheSize number The upper limit of row components kept in the cache, the drawn rows included (by default max(200, 3 × the number of drawn rows)). When it is exceeded, the off-screen rows not seen for the longest time are disposed.
autoRefresh boolean When the reactive data read inside dataRequest changes, it fetches the loaded range again in a single request (on by default); see Automatic refresh. With false, the list refreshes only with refresh().
watch () => unknown An extra dependency for automatic refresh: reactive fields read here trigger a reload too.
autoLoad boolean Asks for the next page when nearing the bottom (on by default). The next page is also asked for when the drawn window comes within overscan of the end of the loaded data (e.g. when the first page does not fill the visible area).
autoLoadThreshold number How many pixels before the bottom edge of the loaded data the next page is asked for (100 by default). With a fixed height, the bottom edge is data.length × itemHeight.
className string A class added to the root div.
style Partial<CSSStyleDeclaration> Inline style for the root div. It is applied after the list's own overflow-y, height and position values; if you give the same property, yours wins.
overscanPages, renderMode, pageBuffer Reserved; they are not read.

The dataRequest contract

dataRequest receives a request object (VirtualizationDataRequest):

Field Type Meaning
page number The zero-based page number.
pageSize number The number of items asked for. In an automatic refresh the loaded range is asked for in a single request, so page is 0 and pageSize is pageSize × the number of loaded pages.
scrollTop number The scroll position at the time of the request (0 on the first load).

and gives a Promise that returns a response object (VirtualizationDataResponse<T>):

Field Type Meaning
items T[] The items on this page.
totalCount number The number of items in the whole data set. It is written to state.totalCount (e.g. for showing the total in a header) and decides the size of the scroll area while hasMore holds.
hasMore boolean Whether there is a next page. When it returns false, automatic loading stops.

The scroll area covers the whole totalCount: items that are not loaded yet take up space in the bottom spacer, at their full height with a fixed itemHeight, and at the average of the loaded rows with a height given as a function. When you scroll into the unloaded region, pages are asked for in order until they reach it. When hasMore becomes false, or totalCount is less than the loaded items, the area is calculated from the loaded data only. filter is applied to the loaded items; the unloaded part is counted unfiltered, and the area shrinks as the items the filter drops are loaded.

Automatic refresh — autoRefresh

autoRefresh (on by default) calls dataRequest inside an effect on the first load and on refresh(). When one of the reactive fields read in this call changes, the loaded range (from page 0 to the current page) is fetched again in a single request; the rows of items still found in the response are kept, and those of the missing ones are disposed (see The rendering model). Next-page requests are not tracked; setData() stops the tracking. In the example, notes.list(folder, state.query) reads the notes array, the search text and the archive, pin and date fields of the notes (and the title and body too while the search is filled in); when one of them changes, the list refreshes.

The compiler wraps the awaits in an async dataRequest function written on the tag: reactive fields read after an await become dependencies too (elsewhere, reads after await are not tracked; see Reads after await). If dataRequest is defined outside the tag, or a dependency comes from somewhere else, give an extra dependency with watch={() => state.query}.

The compiler applies the await wrapping only to a dataRequest written on a <Virtualization> tag; without JSX, only the reactive fields read before the first await become dependencies (see Reads after await). A dependency read after an await or coming from somewhere else is given with watch: () => state.query.

watch is called once while tracking is being set up, inside the same effect as dataRequest; the fields it reads are added to the dependencies, and an error it throws is reported as MJX205. The reload happens on the microtask after the change; the content stays in place until the response arrives. If more than 30 reloads are triggered within a second (loop protection), tracking is stopped and an MJX206 warning is written in development mode; refresh() sets the tracking up again.

Templates and the list state

mainTemplate and loadingTemplate receive the list's state (VirtualizationState<T>). The state fields are reactive; a field read through a getter in a template stays up to date. The data field announces that the loaded array, or the array given with setData, has changed, but it does not wrap the items (see Data is used as given).

Field Type Meaning
isLoading boolean The first load or a refresh() request is in progress.
isInitialized boolean The first response has arrived, or setData() was called.
currentPage number The last loaded page.
totalCount number The totalCount from the last response.
hasMore boolean The hasMore from the last response.
error Error \| undefined The error of the last request; cleared on a successful load.
data T[] The loaded items (with no filter applied), as they were given.

Data is used as given

The list does not turn the data into reactive objects. The items returned by dataRequest or given to setData reach itemTemplate, itemHeight and filter as they are; the very array given to setData is read as state.data and getState().data. Reactive data stays reactive, and plain data stays plain.

A row built with a plain object does not show a value written to the object later on its own. If the row needs to follow the change, give the data with reactive; with plain data, the change is brought to the screen by running the row's bindings again with setState() (see Where reactivity meets the DOM). If a new array has arrived, call setData.

Later pages are not appended to the first page's array: the list collects the loaded items in its own array, and the arrays returned by dataRequest do not change.

A custom container — mainTemplate

mainTemplate builds the layout that wraps the list. Its first parameter (content) is the component that carries the rows; wherever it is put in the layout, the rows are drawn there. The nearest element wrapping it becomes the scroll container: the list gives it overflow-y: auto; position: relative; overflow-anchor: none and listens to its scrolling, while you give its height. In this case no style is written on the root div. A list with a header that shows the folder name and the total number of notes:

<Virtualization<Note>
  itemHeight={64}
  dataRequest={async ({ page, pageSize }) => {
    const found = notes.list(folder, state.query);
    const start = page * pageSize;
    return { items: found.slice(start, start + pageSize), totalCount: found.length, hasMore: start + pageSize < found.length };
  }}
  itemTemplate={(note) => <NoteRow note={note} folder={folder} />}
  mainTemplate={(content, status) => (
    <div class="list">
      <div class="list__head">
        <h2>{FOLDERS[folder].title}</h2>
        <span class="badge">{() => status.totalCount}</span>
      </div>
      <div class="rows">
        <content></content>
      </div>
    </div>
  )}
/>
const list = new Virtualization<Note>({
  itemHeight: 64,
  dataRequest: async ({ page, pageSize }) => {
    const found = notes.list(folder, state.query);
    const start = page * pageSize;
    return { items: found.slice(start, start + pageSize), totalCount: found.length, hasMore: start + pageSize < found.length };
  },
  itemTemplate: (note) => motifComponent(NoteRow, { note, folder }),
  mainTemplate: (content, status) => motifComponent("div", (box: ComponentBase) => {
    box.class.add("list");
    const head = motifComponent("div", (head: ComponentBase) => {
      head.class.add("list__head");
      const title = motifComponent("h2", (h2: ComponentBase) => h2.setText(FOLDERS[folder].title));
      const badge = motifComponent("span", (span: ComponentBase) => {
        span.class.add("badge");
        span.bindings.text(() => `${status.totalCount}`);
      });
      head.controls.add(title, badge);
    });
    const rows = motifComponent("div", (rows: ComponentBase) => {
      rows.class.add("rows");
      rows.controls.add(content);
    });
    box.controls.add(head, rows);
  }),
});
<Virtualization
  itemHeight={64}
  dataRequest={async ({ page, pageSize }) => {
    const found = notes.list(folder, state.query);
    const start = page * pageSize;
    return { items: found.slice(start, start + pageSize), totalCount: found.length, hasMore: start + pageSize < found.length };
  }}
  itemTemplate={(note) => <NoteRow note={note} folder={folder} />}
  mainTemplate={(content, status) => (
    <div class="list">
      <div class="list__head">
        <h2>{FOLDERS[folder].title}</h2>
        <span class="badge">{() => status.totalCount}</span>
      </div>
      <div class="rows">
        <content></content>
      </div>
    </div>
  )}
/>
const list = new Virtualization({
  itemHeight: 64,
  dataRequest: async ({ page, pageSize }) => {
    const found = notes.list(folder, state.query);
    const start = page * pageSize;
    return { items: found.slice(start, start + pageSize), totalCount: found.length, hasMore: start + pageSize < found.length };
  },
  itemTemplate: (note) => motifComponent(NoteRow, { note, folder }),
  mainTemplate: (content, status) => motifComponent("div", (box) => {
    box.class.add("list");
    const head = motifComponent("div", (head) => {
      head.class.add("list__head");
      const title = motifComponent("h2", (h2) => h2.setText(FOLDERS[folder].title));
      const badge = motifComponent("span", (span) => {
        span.class.add("badge");
        span.bindings.text(() => `${status.totalCount}`);
      });
      head.controls.add(title, badge);
    });
    const rows = motifComponent("div", (rows) => {
      rows.class.add("rows");
      rows.controls.add(content);
    });
    box.controls.add(head, rows);
  }),
});

The <content></content> tag places the content parameter.

content is a component, so it is added to the scroll container's controls.

mainTemplate is called once, in the list's constructor; the first request is made when content starts being built.

Filtering

The list can be filtered in two places:

Variable height and size changes

If the rows have different heights that can be calculated in advance, give itemHeight a function; for example (note) => (note.body ? 72 : 48) if the row of a note with a body is taller. The heights are summed once every time the data changes (a load, setData, filter); the visible range is found with a binary search, and scrollToIndex uses these totals as well. Invalid values (NaN, negative) count as 0.

The list does not track heights: even if the value itemHeight reads is reactive, it does not notice the change on its own. When the data stays the same but the height of one or more rows changes (a row expanded, the density setting changed, a fixed itemHeight number was replaced with another number), call updateHeights(). The method reads the heights again from scratch and redraws the window; the rows stay as they are, filter does not run again, and no load is requested. The row at the top of the screen keeps its place on screen, as on a data change: if a row above it grows, the scroll position shifts by the same amount. If a height can only be known by measuring after drawing (a feed of free-form text), this method is not suitable.

When the size of the list's container changes (a screen rotation, the mobile keyboard opening, the window size), the visible rows are recalculated without waiting for a scroll. A ResizeObserver is used for this; if the environment does not have one, this step is skipped, and the observer is released when the list is disposed.

The rendering model and methods

Method Effect
refresh() Resets the data and asks for page 0 again with dataRequest; all rows in the cache are disposed. Sets up automatic refresh tracking again. Returns a Promise.
setData(items) Changes the data without making a request; hasMore becomes false, the page becomes 0, and automatic refresh stops. The rows of items still found in the new data are kept, and those of the missing ones are disposed.
updateHeights() When the data stays the same but heights change, reads the heights again and draws the window; the rows are kept, and the row at the top of the screen stays in place (see Variable height).
scrollToIndex(i) Scrolls to the position of row i (i × itemHeight with a fixed height) and draws right away; an index out of range is clamped to the edge.
getState() A shallow copy of the list state.

The methods are on the list component itself; in JSX you reach it through ref (see Element access inside a component).

The methods are called on the component built with new Virtualization(...).

When to use it?

Situation Recommendation
Few items (< ~100), variable height The normal list binding (Conditional Rendering and Lists).
Many items, a fixed or precomputable row height Virtualization (itemHeight as a number or a function).
Infinite scrolling / a paged API Virtualization + autoLoad.

Next step

To learn memory management and the disposal pattern, go on to Memory Management and Disposal.