How to use it

9 minute read

The Semantic Interoperability Component provides a REST API.

Metadata registry

The metadata of the data sources and agents is managed automatically by the Data Access Manager when an agent is added or removed. The management of agents and data sources is explained in the Data Access Component user guide. The metadata registry is managed through the Orion Context Broker. It is important to note that, while the metadata related with the publish/subscribe agents is managed automatically, the metadata of the on-demand agents (and their corresponding data sources if there is no publish/subscribe agent for the data source) must be registered calling the Orion Context Broker API.

The queries to the Metadata Registry must use the header “Fiware-Service” with the value “metadata”.

Examples:

• Get data sources: GET /v2/entities?type=DataSource&options=keyValue Fiware-Service: metadata
• Get on-demand agents: GET /v2/entities?type=AgentImage&options=keyValues Fiware-Service: metadata
• Get publish/subscribe agents: GET /v2/entities?type=AgentContainer&options=keyValues Fiware-Service: metadata
• Get on-demand agents associated with a data source: GET /v2/entities?type=AgentImage&q=refDataSource==urn:ngsi-ld:DataSource:ExampleDataSource&options=keyValues Fiware-Service: metadata
• Delete data source metadata: DELETE /v2/entities/urn:ngsi-ld:DataSource:ExampleDataSource Fiware-Service: metadata

Data source metadata is stored as DataSource entities. A DataSource entity has the following attributes:

Example of DataSource entity creation:

POST /v2/entities
Fiware-Service: metadata
Content-Type: application/json

{
    "id": "urn:ngsi-ld:ExampleProvider:WeatherObserved",
    "type": "DataSource",
    "attributes": {},
    "dataProvided": {
        "type": "StructuredValue",
        "value": {
            "dateObserved": "string",
            "precipitation": "number",
            "atmosphericPressure": "number",
            "solarRadiation": "number",
            "illuminance": "number",
            "dewPoint": "number",
            "streamGauge": "number",
            "snowHeight": "number",
            "type": "WeatherObserved"
        },
        "metadata": {}
    },
    "description": {
        "type": "Text",
        "value": "WeatherObserved data source",
        "metadata": {}
    },
    "onChain": {
        "type": "Boolean",
        "value": false,
        "metadata": {}
    },
    "dataModels": {
        "type": "Text",
        "value": "https://fiware.github.io/data-models/specs/Weather/WeatherObserved/schema.json",
        "metadata": {}
    }
}

Example of DataSource entity with filtering parameters (defined under “attributes” and “mapping”):

{
    "id": "urn:ngsi-ld:ITI:Customs",
    "type": "DataSource",
    "attributes": {
		"type": "StructuredValue",
		"value": {
			"START_YEAR": "number",
			"MONTH": "number"
		},
		"metadata": {}
	},
	"mapping": {
		"type": "StructuredValue",
		"value": {
			"year": "START_YEAR",
			"month": "MONTH"
		},
		"metadata": {}
	},
    "dataProvided": {
        "type": "StructuredValue",
        "value": {
            "id": "string",
			"customsProcedureID": "string",
			"declaringParty": "object",
			"exportingParty": "object",
			"importingParty": "object",
			"depotParty": "object",
			"customsOffice": "object",
			"customsLocation": "object",
			"customsProvince": "object",
			"customs": "string",
			"customsRegime": "string",
			"customsStatus": "string",
			"customsRegimeRequested": "number",
			"customsDocumentGrossWeight": "number",
			"customsDocumentType": "number",
			"customsDocumentAdmissionDate": "string",
			"customDocumentOriginCountry": "string",
			"customDocumentOriginProvince": "number",
			"customDocumentDestinationCountry": "string",
			"customDocumentDestinationProvince": "number",
			"exportLocation": "object",
			"year": "number",
			"month": "number",
			"TARIC": "number",
			"TARIC_N4": "number",
			"packageType": "string",
			"packageQuantity": "number",
			"portIndex": "number",
			"origin": "object",
			"originExpeditionCountry": "string",
			"destination": "object",
			"conveyor": "object",
			"hasPredeclaration": "boolean",
			"exitType": "string",
			"portCallReference": "number",
			"automaticTranshipment": "boolean",
			"seal": "array",
			"precedingRegimeRequested": "number",
			"unitNumber": "number",
			"statisticalValue": "number",
			"invoiceValue": "number",
			"countryCurrency": "string",
			"container": "number",
			"containerNumber": "string",
			"containerType": "string",
			"containerOperationType": "string",
			"containerNextPrevLoadDischargePortCode": "string",
			"containerGrossWeight": "number",
			"containerDeliveryPlaceCode": "string",
			"containerDischargePortCode": "string",
			"isVGM": "boolean",
			"fullEmpty": "string",
			"transportRegime": "string",
			"transportModeOnBorder": "number",
			"countryTransportMode": "number",
			"transportNationality": "string",
			"exchangeZone": "string",
			"transactionNature": "number",
			"deliveryConditions": "string",
			"contingent": "number",
			"tariffPreference": "number",
			"vesselFreight": "number",
			"fiscalAddressProvince": "number",
			"additionalCodes": "string",
			"type": "Customs"
        },
        "metadata": {}
    },
    "description": {
        "type": "Text",
        "value": "AEAT Customs data source",
        "metadata": {}
    },
    "onChain": {
        "type": "Boolean",
        "value": false,
        "metadata": {}
    },
    "dataModels": {
        "type": "Text",
        "value": "Customs/Customs/schema.json",
        "metadata": {}
    }
}

Agent metadata is stored as AgentImage or AgentContainer entities, for on-demand and publish/subscribe agents, respectively.

An AgentImage entity represents an on-demand agent and has the following attributes:

An AgentContainer entity represents a running publish/subscribe agent and has the following attributes:

Example of AgentImage entity (on-demand agent) creation:

POST /v2/entities
Fiware-Service: metadata
Content-Type: application/json

{
    "id": "urn:ngsi-ld:AgentImage:example-agent",
    "type": "AgentImage",
    "name": {
        "type": "Text",
        "value": "dataportsh2020/example-ondemand"
    },
    "agentType": {
        "type": "Text",
        "value": "on_demand"
    },
	"accessURL": {
        "type":"Text",
        "value":"http://localhost:3000/on_demand"
    },
	"refDataSource": {
		"type": "Relationship",
		"value": "urn:ngsi-ld:ExampleProvider:WeatherObserved"
	}

}

Subscriptions

Publish/subscribe access to the data is provided by the Orion Context Broker.

The main attributes of a subscription are the following:

• “subject”: defines the condition for triggering the notification. It is a JSON object and has the following main attributes:
  • “entities”: Id or type of the entities we want to subscribe to. This object contains two attributes: type and id or idPattern (a regular expression) of the entities of interest.
  • “condition.attrs”: list of attributes that will trigger a notification when changed or created (by default, any changed attribute triggers a notification).
• “notification”: defines how the notifications will be sent. It is a JSON object and has the following main attributes:
  • “http.url”: URL where the notifications will be sent.
  • “attrs”: defines which attributes will be sent in the notification (by default it sends all the attributes).
• “expires” (optional): date and time of expiration for the subscription.

Example of subscription using type and idPattern:

POST /v2/subscriptions

{
 "subject":
   {
   "entities":[{"idPattern":"urn:ngsi-ld:ExampleProvider:WeatherObserved:[A-Za-z0-9]+","type":"WeatherObserved"}]
   },
 "notification":
   {
   "http":{"url":"<MY_CALLBACK_URL>"}
   }
}

Where <MY_CALLBACK_URL> is the URL where the application that consumes the data is listening. You can find more information in the Orion Context Broker documentation

Although subscriptions are mainly intended for the data, it is also possible to subscribe to the metadata registry. In that case, the header “Fiware-Service” with the value “metadata” must be included in the request.

Last values

These requests are provided by the On Demand Component API. The general parameters of these requests are the following:

• “id”: unique identifier of the data source.
• “type”: requested data type.
• “service”: Fiware service value of the data. Corresponds with the “service” attribute of the data source (if it is different from the default value).
• “servicePath”: Fiware servicePath value of the data. Corresponds with the “servicePath” attribute of the data source (if it is different from the default value).
• “attributes”: any parameters for filtering the data. The allowed parameters for a data source are described in the “attributes” field of the DataSource entity and their correspondence with the parameters of the agent is described in the “mapping” field (if these fields are empty or missing in the data source it means that no filtering attributes are allowed for that data source).

Last values requests return the data in the HTTP response message. These requests rely on the information stored in the Metadata Registry. If the request parameters cannot be correctly interpreted according to the registered metadata, the module will return a 400 error code. In addition, this request requires by default the use of an authentication header containing a token issued by the Identity Manager (Keycloak).

Example of last values request:

POST /current
Authorization: Bearer <token>

{
    "id": "urn:ngsi-ld:DataSource:PortCall"
}

Historical data

These requests are provided by the On Demand Component API. The general parameters of these requests are the following:

• “id”: unique identifier of the data source.
• “type”: requested data type.
• “service”: Fiware service value of the data. Corresponds with the “service” attribute of the data source (if it is different from the default value).
• “servicePath”: Fiware servicePath value of the data. Corresponds with the “servicePath” attribute of the data source (if it is different from the default value).
• “callback”: callback URL to receive the historical data.
• “attributes”: any parameters for filtering the data. The allowed parameters for a data source are described in the “attributes” field of the DataSource entity and their correspondence with the parameters of the agent is described in the “mapping” field (if these fields are empty or missing in the data source it means that no filtering attributes are allowed for that data source).

The historical data requests rely on the information stored in the Metadata Registry. If the request parameters cannot be correctly processed according to the registered metadata, the module will return a 400 error code. In addition, this request requires by default the use of an authentication header containing a token issued by the Identity Manager (Keycloak).

Due to the expected data volumes (and delays), the data is sent to the provided callback URL. The response to this request will be 200 OK if the request was processed correctly, otherwise it will return an error code.

Example of historical data request:

POST /historical
Authorization: Bearer <token>

{
    "id": "urn:ngsi-ld:DataSource:PortCall",
    "callback": "<MY_CALLBACK_URL>"
}

Where <MY_CALLBACK_URL> is the URL where the application that consumes the data is listening.

Example of historical data request with filtering parameters:

POST /historical
Authorization: Bearer <token>

{
	"id": "urn:ngsi-ld:ITI:Customs",
	"callback": "<MY_CALLBACK_URL>",
    "attributes": {
        "year": 2022,
        "month": 11
	}
}