How to use it

User guide to understand how the software is used

9 minute read

The Semantic Interoperability Component provides a REST API.

Metadata registry

The metadata of the data sources and agents is managed automatically by the Data Access Manager when an agent is added or removed. The management of agents and data sources is explained in the Data Access Component user guide. The metadata registry is managed through the Orion Context Broker. It is important to note that, while the metadata related with the publish/subscribe agents is managed automatically, the metadata of the on-demand agents (and their corresponding data sources if there is no publish/subscribe agent for the data source) must be registered calling the Orion Context Broker API.

The queries to the Metadata Registry must use the header “Fiware-Service” with the value “metadata”.

Examples:

  • Get data sources: GET /v2/entities?type=DataSource&options=keyValue Fiware-Service: metadata
  • Get on-demand agents: GET /v2/entities?type=AgentImage&options=keyValues Fiware-Service: metadata
  • Get publish/subscribe agents: GET /v2/entities?type=AgentContainer&options=keyValues Fiware-Service: metadata
  • Get on-demand agents associated with a data source: GET /v2/entities?type=AgentImage&q=refDataSource==urn:ngsi-ld:DataSource:ExampleDataSource&options=keyValues Fiware-Service: metadata
  • Delete data source metadata: DELETE /v2/entities/urn:ngsi-ld:DataSource:ExampleDataSource Fiware-Service: metadata

Data source metadata is stored as DataSource entities. A DataSource entity has the following attributes:

Attribute name type description mandatory
id text Unique id of the data source (urn format). The DataSource id has the same value as the common part of the entityId. Yes
type text Predefined value for the entity type that represents a data source (“DataSource”) Yes
description text Human-readable description of the data source No
dataProvided object Description of the entities provided by this data source. It represents the attributes provided by this data source. This object contains an attribute named “type”, which provides the value of the entity type of the data provided by the data source. Yes
dataModels text Reference to the class of the Data Model (URL of the JSON schema) Yes
service text Fiware Service value for the data of this data source (if different from the default value) No
servicePath text Fiware ServicePath value for the data of this data source (if different from the default value) No
onChain boolean Defines if the data is stored locally or on Blockchain. (Default: false). No
attributes object JSON object describing the attributes defined in the data source that could be used in the requests to the On Demand component. These attributes are defined as parameters in the agent. Each attribute is defined as “name”:“data type”. If no parameters can be used for filtering, this object will be empty. No
mapping object Defines the mapping between the attributes allowed in the requests to the On Demand component and the parameters of the agent listed in “attributes”. If possible, it is recommended to use the same names in the request parameters as in the corresponding attributes defined in the Data Model (it may not be possible in all cases; for instance, if the data model defines a “date” attribute and the data source allows filtering by a time period). If no parameters can be used for filtering, this object will be empty. No

Example of DataSource entity creation:

POST /v2/entities
Fiware-Service: metadata
Content-Type: application/json

{
    "id": "urn:ngsi-ld:ExampleProvider:WeatherObserved",
    "type": "DataSource",
    "attributes": {},
    "dataProvided": {
        "type": "StructuredValue",
        "value": {
            "dateObserved": "string",
            "precipitation": "number",
            "atmosphericPressure": "number",
            "solarRadiation": "number",
            "illuminance": "number",
            "dewPoint": "number",
            "streamGauge": "number",
            "snowHeight": "number",
            "type": "WeatherObserved"
        },
        "metadata": {}
    },
    "description": {
        "type": "Text",
        "value": "WeatherObserved data source",
        "metadata": {}
    },
    "onChain": {
        "type": "Boolean",
        "value": false,
        "metadata": {}
    },
    "dataModels": {
        "type": "Text",
        "value": "https://fiware.github.io/data-models/specs/Weather/WeatherObserved/schema.json",
        "metadata": {}
    }
}

Example of DataSource entity with filtering parameters (defined under “attributes” and “mapping”):

{
    "id": "urn:ngsi-ld:ITI:Customs",
    "type": "DataSource",
    "attributes": {
		"type": "StructuredValue",
		"value": {
			"START_YEAR": "number",
			"MONTH": "number"
		},
		"metadata": {}
	},
	"mapping": {
		"type": "StructuredValue",
		"value": {
			"year": "START_YEAR",
			"month": "MONTH"
		},
		"metadata": {}
	},
    "dataProvided": {
        "type": "StructuredValue",
        "value": {
            "id": "string",
			"customsProcedureID": "string",
			"declaringParty": "object",
			"exportingParty": "object",
			"importingParty": "object",
			"depotParty": "object",
			"customsOffice": "object",
			"customsLocation": "object",
			"customsProvince": "object",
			"customs": "string",
			"customsRegime": "string",
			"customsStatus": "string",
			"customsRegimeRequested": "number",
			"customsDocumentGrossWeight": "number",
			"customsDocumentType": "number",
			"customsDocumentAdmissionDate": "string",
			"customDocumentOriginCountry": "string",
			"customDocumentOriginProvince": "number",
			"customDocumentDestinationCountry": "string",
			"customDocumentDestinationProvince": "number",
			"exportLocation": "object",
			"year": "number",
			"month": "number",
			"TARIC": "number",
			"TARIC_N4": "number",
			"packageType": "string",
			"packageQuantity": "number",
			"portIndex": "number",
			"origin": "object",
			"originExpeditionCountry": "string",
			"destination": "object",
			"conveyor": "object",
			"hasPredeclaration": "boolean",
			"exitType": "string",
			"portCallReference": "number",
			"automaticTranshipment": "boolean",
			"seal": "array",
			"precedingRegimeRequested": "number",
			"unitNumber": "number",
			"statisticalValue": "number",
			"invoiceValue": "number",
			"countryCurrency": "string",
			"container": "number",
			"containerNumber": "string",
			"containerType": "string",
			"containerOperationType": "string",
			"containerNextPrevLoadDischargePortCode": "string",
			"containerGrossWeight": "number",
			"containerDeliveryPlaceCode": "string",
			"containerDischargePortCode": "string",
			"isVGM": "boolean",
			"fullEmpty": "string",
			"transportRegime": "string",
			"transportModeOnBorder": "number",
			"countryTransportMode": "number",
			"transportNationality": "string",
			"exchangeZone": "string",
			"transactionNature": "number",
			"deliveryConditions": "string",
			"contingent": "number",
			"tariffPreference": "number",
			"vesselFreight": "number",
			"fiscalAddressProvince": "number",
			"additionalCodes": "string",
			"type": "Customs"
        },
        "metadata": {}
    },
    "description": {
        "type": "Text",
        "value": "AEAT Customs data source",
        "metadata": {}
    },
    "onChain": {
        "type": "Boolean",
        "value": false,
        "metadata": {}
    },
    "dataModels": {
        "type": "Text",
        "value": "Customs/Customs/schema.json",
        "metadata": {}
    }
}

Agent metadata is stored as AgentImage or AgentContainer entities, for on-demand and publish/subscribe agents, respectively.

An AgentImage entity represents an on-demand agent and has the following attributes:

Attribute name type description mandatory
id text Unique identifier of the on-demand agent (urn format) Yes
type text Predefined value for the entity type that represents an agent image (“AgentImage”) Yes
agentType text Type of NGSI Agent for this image (“on_demand”) Yes
name text Name of the Docker image for this agent. Yes
accessURL URL URL for on demand data requests (endpoint of the Data Access Manager) Yes
refDataSource Relationship Unique identifier of the corresponding data source Yes

An AgentContainer entity represents a running publish/subscribe agent and has the following attributes:

Attribute name type description mandatory
id text Unique identifier of the publish/subscribe agent (urn format) Yes
type text Predefined value for the entity type that represents a running publish/subscribe agent (“AgentContainer”) Yes
agentType text Type of NGSI Agent for this image (“publish-subscribe”) Yes
name text Name of the Docker image for this agent. Yes
refDataSource Relationship Unique identifier of the corresponding data source Yes

Example of AgentImage entity (on-demand agent) creation:

POST /v2/entities
Fiware-Service: metadata
Content-Type: application/json

{
    "id": "urn:ngsi-ld:AgentImage:example-agent",
    "type": "AgentImage",
    "name": {
        "type": "Text",
        "value": "dataportsh2020/example-ondemand"
    },
    "agentType": {
        "type": "Text",
        "value": "on_demand"
    },
	"accessURL": {
        "type":"Text",
        "value":"http://localhost:3000/on_demand"
    },
	"refDataSource": {
		"type": "Relationship",
		"value": "urn:ngsi-ld:ExampleProvider:WeatherObserved"
	}

}

Subscriptions

Publish/subscribe access to the data is provided by the Orion Context Broker.

The main attributes of a subscription are the following:

  • “subject”: defines the condition for triggering the notification. It is a JSON object and has the following main attributes:
    • “entities”: Id or type of the entities we want to subscribe to. This object contains two attributes: type and id or idPattern (a regular expression) of the entities of interest.
    • “condition.attrs”: list of attributes that will trigger a notification when changed or created (by default, any changed attribute triggers a notification).
  • “notification”: defines how the notifications will be sent. It is a JSON object and has the following main attributes:
    • “http.url”: URL where the notifications will be sent.
    • “attrs”: defines which attributes will be sent in the notification (by default it sends all the attributes).
  • “expires” (optional): date and time of expiration for the subscription.

Example of subscription using type and idPattern:

POST /v2/subscriptions

{
 "subject":
   {
   "entities":[{"idPattern":"urn:ngsi-ld:ExampleProvider:WeatherObserved:[A-Za-z0-9]+","type":"WeatherObserved"}]
   },
 "notification":
   {
   "http":{"url":"<MY_CALLBACK_URL>"}
   }
}

Where <MY_CALLBACK_URL> is the URL where the application that consumes the data is listening. You can find more information in the Orion Context Broker documentation

Although subscriptions are mainly intended for the data, it is also possible to subscribe to the metadata registry. In that case, the header “Fiware-Service” with the value “metadata” must be included in the request.

Last values

These requests are provided by the On Demand Component API. The general parameters of these requests are the following:

  • “id”: unique identifier of the data source.
  • “type”: requested data type.
  • “service”: Fiware service value of the data. Corresponds with the “service” attribute of the data source (if it is different from the default value).
  • “servicePath”: Fiware servicePath value of the data. Corresponds with the “servicePath” attribute of the data source (if it is different from the default value).
  • “attributes”: any parameters for filtering the data. The allowed parameters for a data source are described in the “attributes” field of the DataSource entity and their correspondence with the parameters of the agent is described in the “mapping” field (if these fields are empty or missing in the data source it means that no filtering attributes are allowed for that data source).

Last values requests return the data in the HTTP response message. These requests rely on the information stored in the Metadata Registry. If the request parameters cannot be correctly interpreted according to the registered metadata, the module will return a 400 error code. In addition, this request requires by default the use of an authentication header containing a token issued by the Identity Manager (Keycloak).

Example of last values request:

POST /current
Authorization: Bearer <token>

{
    "id": "urn:ngsi-ld:DataSource:PortCall"
}

Historical data

These requests are provided by the On Demand Component API. The general parameters of these requests are the following:

  • “id”: unique identifier of the data source.
  • “type”: requested data type.
  • “service”: Fiware service value of the data. Corresponds with the “service” attribute of the data source (if it is different from the default value).
  • “servicePath”: Fiware servicePath value of the data. Corresponds with the “servicePath” attribute of the data source (if it is different from the default value).
  • “callback”: callback URL to receive the historical data.
  • “attributes”: any parameters for filtering the data. The allowed parameters for a data source are described in the “attributes” field of the DataSource entity and their correspondence with the parameters of the agent is described in the “mapping” field (if these fields are empty or missing in the data source it means that no filtering attributes are allowed for that data source).

The historical data requests rely on the information stored in the Metadata Registry. If the request parameters cannot be correctly processed according to the registered metadata, the module will return a 400 error code. In addition, this request requires by default the use of an authentication header containing a token issued by the Identity Manager (Keycloak).

Due to the expected data volumes (and delays), the data is sent to the provided callback URL. The response to this request will be 200 OK if the request was processed correctly, otherwise it will return an error code.

Example of historical data request:

POST /historical
Authorization: Bearer <token>

{
    "id": "urn:ngsi-ld:DataSource:PortCall",
    "callback": "<MY_CALLBACK_URL>"
}

Where <MY_CALLBACK_URL> is the URL where the application that consumes the data is listening.

Example of historical data request with filtering parameters:

POST /historical
Authorization: Bearer <token>

{
	"id": "urn:ngsi-ld:ITI:Customs",
	"callback": "<MY_CALLBACK_URL>",
    "attributes": {
        "year": 2022,
        "month": 11
	}
}

Last modified April 19, 2023: Files update (a811bf0)