The current options to handle the REST API response data are cumbersome. Most of the time, the API Consumer receives a large set of unwanted data, mainly in a format that requires further handling before the target system or App can consume the data.
There is generally no option to only fetch the specific fields required for the consuming system. The consumer downloads all the fields from the data model and has to parse the response data to extract the required fields. This becomes a high-impact issue with extensive data set as it increases the consumer system’s response time.
Having to download a large set of data records is also a common issue faced by API consumers. There is generally no control on limiting the number of records to download.
In other complex scenarios, a Consumer looking to have data for any child or linked records will have to execute multiple API calls and combine the data to achieve the target payload before it can be consumed. This introduces a complex API consumption pattern and adds to the development timelines.
Additionally, in scenarios where the Consumer system needs to download the media like documents, images, etc., a separate API call is required to fetch each media record for a parent record. This again creates a complex API consumption pattern and degrades the performance considerably for the Consumer system.
These issues and gaps were on top of our minds while looking at ways to control the response output and only fetch the required data in the required format.
Hint Parameter (represented by “h”):
We created a “Hint Parameter” option to allow the Consumer to control the handling of the response data.
The Hint Parameter consists of the following Operators:
Data Control Operator - allows tighter control on handling the data. For example, “Sort” can be used to sort the data with one or more fields; “Max” can be used to limit the no of records being returned; “Fields” can be used to select only specific fields to be returned; “Page” allows to paginate between the set of records.
Child & Linked Data Operator - allows pre-fetching of a child or linked records from within the parent record. For example, “Children” can fetch records from hierarchy objects. A use case could be to pre-fetch all the floors for a Building record. Similarly, “Linked” can fetch records from linked objects. A use case could be to pre-fetch linked records like documents, comments, images, etc.
Media Operator - allows pre-fetching of the content of particular pre-defined linked objects. For example, “Media” can be used to download the content for documents or images along with the parent record. As a default behaviour, only metadata is included in the payload for the media fields like documents, images, etc. This operator can help skip multiple costly calls to the backend APIs.
How do these operators work?
All operators become part of the query string in a pre-defined structure. The hint parameter has a JSON syntax and can apply complex controls on the response data.
Let’s take a use case:
A user would like to sort the results by the fields - postcode and city.
The hint parameter (represented by “h”) will be like this:
h={"_sort": [{"postCode":"_asc"},{"city": "_desc"}]}Similarly, if we only need specific fields to be retrieved from the data model, the hint parameter will be like this:
h={"_fields":["name", "city"]}The API Explorer tool has all these operators available through a friendly UI to select and construct the hint you want to use.
Data Control Operators:
Data Control Operators are a group of operators which allow you to control the outcome of the response data. These hint operators are applied to the results extracted from the data store before the data is transmitted from the server.
These control help prevent unwanted data downloads, which helps improve the transmission time and data parsing performance.
Sort:
h={"_sort": [{"postCode":"_asc"}, {"city":"_desc"}]}No. of Records:
h={"_max": 10}Required Fields:
h={"_fields": ["name":"city"]}Pagination:
h={"_page": [{"_size":100}, {"_offset":0}]}The Sort operator can be used for sorting the results by using one or more fields in a combination of ascending or descending order.
The Max operator can limit the number of records being returned.
The Fields operator can filter the fields being returned. If the API payload has many fields and only a few fields are required, this filtering can be applied on the server preventing unnecessary data transmission.
The Pagination operator can paginate between sets of pages within a result set. The Size keyword allows control of the number of records to be returned, and the Offset keyword allows transmission of the records from a specific row onward.
Child & Linked Data Operators:
The Child and Linked Data controllers are helpful in scenarios where the Consumer is working on a record but also needs to display the related records.
The Child operator allows fetching the child records within a hierarchy. For example, if the Consumer system is looking to show/use the Floors while showing the Buildings. Generally, this will require an additional API call for each Building record to fetch its Floors. All this can be accomplished in one call by adding the Child operator. We can either specify a specific child object to be returned or if we are unsure of the child objects, we can use the wildcard ”*” to return all the child objects.
The Linked operator allows the fetching of linked object records. For example, if the Consumer system is looking to fetch Assets, Documents, Comments, etc. while working on a Work task record. Generally, this will require multiple calls to the API, one per object, to achieve the desired results. All this can be accomplished in one call by adding the Linked operator, and as mentioned above, one could use wildcard ”*” to return all the related objects if you are unsure. Do keep in mind that the linked objects come with a subset of fields to maintain the integrity of the API.
Media Operator:
The media operator is helpful in scenarios where the Consumer system wants to download the media content with the metadata.
For example, using the Linked operator, we can get the metadata for the documents. However, if we need to download the actual document or a thumbnail for images, we need to make subsequent calls to the API.
The Media operator allows us to download the content in Base64 encoded format with the core records, thereby improving the overall user experience.
The default behaviour is to skip the content download unless otherwise instructed through the Media operator. On the other hand, independent APIs can download the content separately.