Skip to main content

Incremental fetching

After the initial import, fetch only the records that changed since your previous run. This page describes how to do that without missing changes.

In short
  1. Before the first request of a run, note the current time in UTC. This is the run's start time.
  2. Request the records with updatedInDataMart set to one minute before the start time of your previous successful run.
  3. Retrieve every page, passing the same updatedInDataMart value on every page.
  4. Insert or replace each record in your copy, matched on its natural key. If the query has an includeDeleted argument, pass includeDeleted: true and delete your copy of each record returned with deletedDatabaseRecord: true.
  5. Only when every page is processed, save the start time of this run for the next one.

How the timestamp works​

  • Every record has updatedInDataMartDateTime: the UTC time when Data Mart last wrote it. It changes whenever Data Mart writes the record, for example after a change in Visma.net ERP, a correction, a deletion, or a backfill announced in the release notes.
  • updatedInDataMart returns the records whose timestamp is strictly later than the value you pass. The optional updatedInDataMartTo returns the records whose timestamp is strictly earlier than its value.
  • Results are ordered by id, not by timestamp. Use lastId only to page through the results of one run (see Pagination).

Start from the start time, not from the highest timestamp​

Storing the highest updatedInDataMartDateTime you received and starting the next run from it can miss changes. A run pages through the results by id. A record on a page you have already read can change while the run continues; its id stays the same, so this run does not return it again. A later page can then return a record with a higher timestamp, and a next run that starts from that timestamp skips the earlier change for good.

TimeWhat happens
10:00The run starts. Page 1 returns customer 1001.
10:05Customer 1001 changes in ERP. Data Mart stamps it 10:05.
10:19A later page returns customer 2002, stamped 10:18. The run ends.
Next run, from the highest timestampStarts after 10:17 and never returns the 10:05 change of customer 1001.
Next run, from the start timeStarts after 09:59 and returns customer 1001 with its 10:05 change.

The start time does not have this problem: every record that changes during a run gets a timestamp after that run's start time, so the next run returns it.

Why one minute earlier​

Data Mart writes many records in parallel. Each group of up to 1,000 records gets its timestamp right before it is written, but a group with an earlier timestamp can finish writing shortly after a group with a later one. A run that starts exactly at the previous start time could skip such a record. Starting one minute earlier covers this delay.

The margin also absorbs a small difference between your clock and Data Mart's. Keep your clock synchronized with a time server; one minute does not cover a clock that is off by a minute or more.

Records you receive more than once​

Because of the overlap, each run returns some records you already received in the previous run. Match every record on its natural key, companyId plus the fields marked @partOfKey (see Identifying Entities), and replace your copy. Do not use id to identify records: it is meant for pagination only and can change.

Deleted records​

  • On queries that have an includeDeleted argument, pass includeDeleted: true. It defaults to false, and without it deleted records are not returned at all, so your copy keeps them.
  • A deleted record is returned with deletedDatabaseRecord: true and a new updatedInDataMartDateTime. Delete your copy, matched on the natural key.
  • The overlap can return a deletion again. Deleting a record you already removed has no effect.
  • ERP can delete a record and create it again under the same key. It is then returned as deleted in one run and as a current record in a later run, so process your runs in order.

Queries without an includeDeleted argument do not track deletions (see Deletion support).

When a run fails​

Do not save the start time of a run that did not finish. The next run starts again from the start time you saved before, and returns everything since then. Records you already processed in the failed run are returned again and replace your copy.

The initial import​

Note the start time before the first request of the initial import, and do not pass updatedInDataMart. When every page is imported, save that start time. The first incremental run then starts one minute before it and returns every record that changed while the import was running.

Example​

The previous run started at 2026-10-09T08:00:00Z, so this run passes 2026-10-09T07:59:00Z:

query {
customers(
updatedInDataMart: "2026-10-09T07:59:00Z"
includeDeleted: true
pageSize: 5000
) {
id
companyId
customerId
deletedDatabaseRecord
updatedInDataMartDateTime
customerNumber
customerName
}
}

For the next page, add lastId with the id of the last record on the current page, and keep updatedInDataMart unchanged. Continue until a page returns no records. Queries that page with lastGroupKey instead of lastId follow the same rule for updatedInDataMart.