Spark® API Replication
If you'd like to replicate data but do not have an API key with replication permission, or if you find a problem with this outline, please contact api-support@fbsdata.com.
- Overview
- Step 1. Initial Download and Pagination
- Step 2. Updating Data
- Step 3. Purging Stale Data
- Best Practices
Overview
Replication API Keys
Developers who wish to replicate data need to use an API key with replication access (now the default). API keys with permission to replicate must hit a separate replication endpoint: https://replication.sparkapi.com. Requests from API keys with replication access will fail if they are made to the regular Spark API endpoint.
To make replicating data easier, these keys also have the following privileges:
- The maximum
_limitis raised to1000for listings and accounts, and200for other resources. - The accounts resource does not apply company or office restrictions when searching user records for IDX roles.
- The
CustomFieldsRawandCustomFieldsExpandedRawexpansions are available for listings when retrieving multiple records.
Replication API keys are otherwise identical to regular API keys, with their access to resources and data governed by the API key's role.
The Replication Process
When replicating records from any resource, there are three steps:
Note: Replicate & Use Member and Office Data for Listing Display
We strongly recommmend that in addition to replicating the listings data, data consumers replicate member and office data from the accounts resource and use this data when displaying listings. See Displaying Agent & Office Data on Listings below for more information on why this is important.
While the examples below demonstrate how to duplicate from the listings resource, the same basic approach is applicable for replicating account records.
Step 1. Initial Download and Pagination
To kick off the initial download you'll need to make a request to the all listings endpoint. Be sure to use the _skiptoken parameter to paginate through the results, and a _limit=1000 parameter to return 1000 records at a time.
Optionally, you may include the _select parameter, but no other parameters should be specified.
Important: a ListingKey orderby is applied by default, ensuring that the order of the results
is consistent while paginating with _skiptoken.
GET https://replication.sparkapi.com/v1/listings?_limit=1000&_skiptoken=
Use the value of the _skiptoken parameter returned in each response as the value for your next request. E.g.
GET https://replication.sparkapi.com/v1/listings?_limit=1000&_skiptoken=20000612234845908076000000Incrementing the _skip parameter (use with caution)
An alternative method of paginating through results is using the _skip parameter. This specifies a number of records to skip before returning results. Use this method with caution; see below.
If this is your first request...
GET https://replication.sparkapi.com/v1/listings?_limit=1000...then your second request should include a _skip=1000 parameter...
GET https://replication.sparkapi.com/v1/listings?_limit=1000&_skip=1000...and so on. Increment your _skip value by whatever value you are using for the _limit parameter. Note that this differs from the _skiptoken parameter, which takes a ListingKey as a value and skips all records ahead of that listing.
Please note that the _skiptoken parameter is not implemented for the accounts resource, and the _$skip parameter should be used for agent and office data.
A caveat about using the _skip parameter
Paginating using the _skip parameter can cause you to replicate the same listing more than once unless your process controls for it. For example, if one new listing is entered between when you pull the first and the second pages of results, the last record of the first page will now show up as the first record on the second page, i.e. it was the 1,000th listing when the first API request was made, but now it's the 1,001st listing when the second API request is made. Given this, your process must account for this possibility.
2. Updating Data
Keeping your local data updated involves making periodic requests for recently updated listings, which is done by using the ModificationTimestamp field with the _filter parameter.
We recommend polling for updated records no less than once every hour. to ensure your data reflects recent changes.
To request listings that have been modified within a given window, use a _filter that includes both an lower bound - ModificationTimestamp gt <timestamp> - and an upper bound: ModificationTimestamp lt <timestamp>. You can also use the "between" syntax: here's an example:
GET https://replication.sparkapi.com/v1/listings?_filter=ModificationTimestamp bt 2024-07-01T00:00:00Z,2024-07-02T00:00:00Z&_limit=1000&_skiptoken=
Paginating through the updated results is once again accomplished by using _skiptoken.
During this step it is recommended that you also update any listing subresources of interest. Note that our listing data has several timestamps that can be used to determine if the corresponding data needs to be updated: PhotosChangeTimestamp, VideosChangeTimestamp and DocumentsChangeTimestamp.
Server-side caching
We use server-side caching, which stores for a short time (10min) indexes of listings that match a given filter. If you make the same request twice in a row, one minute apart, the cached index of matching listings dictates which listings will be returned in the second response. Caching is done on a per-API key basis; you'll never pull someone else's cached results list.
To avoid errors caused by caching, always use two timestamps to specify a start and end of your polling window. Polling only based on one ModificationTimestamp - e.g. _filter=ModificationTimestamp gt <timestamp> - can cause errors. For example, if the first request fails, and you successfully retry it a minute later, you will not receive any listings that were modified between your first and second requests.
3. Purging Stale Data
Purging stale data is the final part of the replication process. It's common to lose access to listings that were previously accessible with your API key. This most frequently happens when a listing changes status, and with IDX API keys, though it can happen with API keys of any role. Consequently, it's necessary to have a process to check whether listings in your local data can still be accessed.
This process should be run no less than once daily, and MLS-specific rules may require more frequent updates. Failure to do so can result in continuing to display listings that are no longer active or eligible for display.
We recommend using a _select=ListingKey parameter to limit the data you get back, then paginating through the listings results as outlined above in step 1 (again, the _skiptoken method is recommended for listings). Compare all ListingKey values returned with the values in your local data, and purge any records that do not match.
GET https://replication.sparkapi.com/v1/listings?_limit=1000&_select=ListingKey&_skiptoken=Best Practices
Displaying Agent & Office Data on Listings
We strongly recommend data consumers:
- Display listing and buyer agent and office data from agent and office account records rather than from the listing payloads themselves.
- Replicate and periodically update from the accounts resource to ensure agent and office records remain up-to-date.
While the API response from the listings resource returns listing and buyer agent and office fields (e.g. ListAgentPreferredPhone), agent and office data is pulled in from the associated account records at the time of the request; it is not stored as part of the actual property records. Changes to account records do not change listing record modification timestamps.
The following fields from the listings response contain keys for the corresponding Member and Office records:
ListAgentIdListOfficeIdListCompanyIdCoListAgentIdCoListOfficeIdCoListCompanyIdBuyerAgentIdBuyerOfficeIdBuyerCompanyIdCoBuyerAgentIdCoBuyerOfficeIdCoBuyerCompanyId
For example, if a listing record returns this ListAgentId...
"ListAgentId": 20140227182254339191000000,...then the following request will return the account record with the data that should be displayed:
GET https://replication.sparkapi.com/v1/accounts/20140227182254339191000000Replication Optimizations
We recommend the following best practices to ensure efficient data replication and data accuracy.
Custom fields: By default, you'll receive back the standard listing fields from the listings endpoint. Custom fields are MLS-specific fields, often locally oriented, and they are not included in the default RESO Web API response. You can include them by using an _expand=CustomFields parameter with your request. Only request custom fields back if you need this local data, as this significantly increases the size of the payload.
The _select parameter: Use the _select parameter with your requests to limit the fields you receive back to only those which you actually use. Doing so can cut out large amounts of data you don't need.
Expansions: If you use Photos, Vidoes, Virtual Tours, Rooms, Units, Open Houses, or Documents data, you can request it be included along with the standard response from the listings resource, by including any of the desired expansions with your request:
GET https://replication.sparkapi.com/v1/listings?_limit=1000&_expand=Photos,Videos,,VirtualTours,Rooms,Units,OpenHouses,DocumentsA full list of possible listing expansions can be reviewed here.
Alternatively, you can retrieve this data with separate requests to property-specific endpoints (e.g. /listings/<ListingKey>/photos). If you do so, we strongly encourage you to make use of the PhotosChangeTimestamp, VideosChangeTimestamp, OpenHousesChangeTimestamp, and DocumentsChangeTimestamp fields to see if any media needs to be updated prior to polling for that data.