Commit 055c6917 authored by Jim White's avatar Jim White Committed by Michael Hall

copying doc changes made to master to edinburgh branch

Signed-off-by: default avatarJim White <jpwhite_mn@yahoo.com>
parent 5be7425f
......@@ -45,6 +45,7 @@ COPY getting-started ./
COPY walk-through ./
COPY quick-start ./
COPY security/* ./
COPY application/* ./
COPY entrypoint.sh /entrypoint.sh
......
##################################
Application Services Microservices
##################################
.. image:: ApplicationServices.png
Application Services are the means to extract, process/transform and send event/reading data from EdgeX to an endpoint or process of your choice.
Application Services are based on the idea of a "Functions Pipeline". A functions pipeline is a collection of functions that process messages (in this case EdgeX event/reading messages) in the order that you've specified. The first function in a pipeline is a trigger. A trigger begins the functions pipeline execution. A trigger is something like a message landing in a watched message queue.
An SDK is provided (the Applicaiton Functions SDK) to help build Application Services by assembling triggers, pre-existing functions and custom functions of your making into a pipeline.
.. toctree::
:maxdepth: 1
Ch-ApplicationServices
Ch-ApplicationFunctionsSDK
**Note** Application Services will replace Export Services in a future EdgeX release.
This diff is collapsed.
Application Services
====================
Application Services are a means to get data from EdgeX Foundry to external systems and process (be it analytics package, enterprise or on-prem application, cloud systems like Azure IoT, AWS IoT, or Google IoT Core, etc.). Application Services provide the means for data to be prepared (transformed, enriched, filtered, etc.) and groomed (formatted, compressed, encrypted, etc.) before being sent to an endpoint of choice. Endpoints supported out of the box today include HTTP and MQTT endpoints, but will include additional offerings in the future and could include a custom endpoints.
The intention of Application Services are to address scalability concerns of the existing EdgeX Export Client and Distribution Services (Export Services) as well as provide a flexible solution for exporting data outside of EdgeX without encumbering the EdgeX development community itself with trying to support all major cloud providers and export solutions. For the Edinburgh release cycle, the existing Client and Export Service remain supported and are still considered the primary way to export data out of EdgeX. However, it is encouraged for new development efforts adopting EdgeX that the App Functions SDK and resulting Application Services be leveraged moving forward with the intention that by the Fuji release, the SDK will be moved into release status and become the primary method of exporting data from EdgeX.
Application Services are based on the idea of a "Functions Pipeline". A functions pipeline is a collection of functions that process messages (in this case EdgeX event/reading messages) in the order that you've specified. The first function in a pipeline is a trigger. A trigger begins the functions pipeline execution. A trigger is something like a message landing in a watched message queue.
.. image:: TriggersFunctions.png
An Applications Functions Software Development Kit (or App Functions SDK) is a available to help create Application Services. Currently the only SDK supported language is Golang, with the intention that community developed and supported SDKs will come in the future for other languages. It is currently available as a Golang module to remain operating system (OS) agnostic and to comply with the latest EdgeX guidelines on dependency management.
Export Service Deficiencies
---------------------------
With the current export services, developers register their endpoints or MQTT clients with the provided client registration service and as events are consumed from Core Data, the export service would then relay that information to the registered endpoints in a sequential fashion. Requiring the individual export service to rebroadcast data to all registered endpoints overtime creates a bottleneck and leaves applications with a potential delay in receiving events. Furthermore, the overhead and complexity of managing all registered endpoints becomes an added burden to EdgeX services. Finally, the Export services have also begun to address a bit more than is sustainable in regard to supporting all the major cloud provider endpoints. Providing an SDK and removing cloud specific exports is one way to remain agnostic to cloud providers and enables 3rd parties to support their use of any given cloud solution and eliminates the dependency
on EdgeX to support the ever-changing cloud environment.
Application Service Improvements
---------------------------------
Providing an SDK that connects directly to a message bus by which Core Data events are published eliminates performance issues as well as allow the developers extra control on what happens
with that data as soon as it is available. Furthermore, it emphasizes configuration over registration for consuming the data. The application services can be customized to a client's needs and thereby also removing the need for client registration.
Standard Functions
------------------
As mentioned, an Application Service is a function pipeline. The SDK provides some standard functions that can be used in a functions pipeline. In the future, additional functions will be provided "standard" or in other words provided with the SDK. Additinally, developers can implement their own custom functions and add those to the Application Service functions pipeline.
.. image:: SDKFunctions.png
One of the most common use cases for working with data that come from Core Data is to filter data down to what is relevant for a given application and to format it. To help facilitate this, four primary functions ported over from the existing services today are included in the SDK. The first is the `DeviceNameFilter` function which will remove events that do not match the specified IDs and will cease execution of the pipeline if no event matches. The second is the `ValueDescriptorFilter` which exhibits the same behavior as `DeviceNameFilter` except filtering on Value Descriptor instead of DeviceID. The third and fourth provided functions in the SDK transform the data received to either XML or JSON by calling `XMLTransform` or `JSONTransform`.
Typically, after filtering and transforming the data as needed, exporting is the last step in a pipeline to ship the data where it needs to go. There are two primary functions included in the SDK to help facilitate this. The first is `HTTPPost(string url)` function that will POST the provided data to a specified endpoint, and the second is an `MQTTPublish()` function that will
publish the provided data to an MQTT Broker as specified in the configuration.
There are two primary triggers that have been included in the SDK that initiate the start of the function pipeline. First is via a POST HTTP Endpoint `/trigger` with the EdgeX event data as the body. Second is the MessageBus subscription with connection details as specified in the configuration.
Finally, data may be sent back to the message bus or HTTP response by calling.complete() on the context. If the trigger is HTTP, then it will be an HTTP Response. If the trigger is MessageBus, then it will be published to the configured host and topic.
Examples
--------
There are three example Application Services provided in the app-functions-sdk-go repository in the /examples directory that attempt to show basic structure of building an application with the app functions sdk. They also focus on how to leverage various built in provided functions as mentioned above as well as how to write your own in the case that the SDK does not provide what is needed.
#.
`Simple Filter XML <https://github.com/edgexfoundry/app-functions-sdk-go/tree/master/examples/simple-filter-xml>`_ -> Demonstrates Filter of data by device ID and
transforming data to XML
#.
`Simple Filter XML Post <https://github.com/edgexfoundry/app-functions-sdk-go/tree/master/examples/simple-filter-xml-post>`_ -> Same example as #1, but result published to HTTP
Endpoint
#.
`Simple Filter XML MQTT <https://github.com/edgexfoundry/app-functions-sdk-go/tree/master/examples/simple-filter-xml-mqtt>`_ -> Same example as #1, but result published to MQTT
Broker
The features in the initial implementation of the App Functions SDK should be sufficient to provide the foundation for basic filtering and export needs. There are some functions in the existing export services that are not yet available in application functions and are intended to be included in a later release. This includes the Encryption Transformer, the Compressor Transformer, and Valid Event Check. See `Unsupported existing export service functions`_. The primary purpose for leaving this out was to address core pieces of functionality that would set up the ease of adding additional functions in the future.
Unsupported existing export service functions
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
From :doc:`./Ch-Distribution`:
**Compressor Transformer**\ –A transformer component compresses the data string to be delivered to the clients, for any clients that have requested their data be compressed either by GZIP or ZIP methods.
**Encryption Transformer**\ –An encryption component encrypts the data to be sent to the client, using the client provided keys and vectors.
**Valid Event Check**\ –The first component in the pipe and filter, before the copier (described in the previous section) is a filter that can be optionally turned on or off by configuration. This filter is a general purpose data checking filter which assesses the device- or sensor-provided Event, with associated Readings, and ensures the data conforms to the ValueDescriptor associated with the Readings.
* For example, if the data from a sensor is described by its metadata profile as adhering to a “Temperature” value descriptor of floating number type, with the value between -100° F and 200° F, but the data seen in the Event and Readings is not a floating point number, for example if the data in the reading is a word such as “cold,” instead of a number, then the Event is rejected (no client receives the data) and no further processing is accomplished on the Event by the Export Distro service.
HTTP Trigger
============
Designating an HTTP trigger will allow the pipeline to be triggered by a RESTful POST call to http://[host]:[port]/trigger/. The body of the POST must be an EdgeX event.
edgexcontext.Complete([]byte outputData) - Will send the specified data as the response to the request that originally triggered the HTTP Request.
In the main() function, note the call to HTTPPostXML or HTTPPostJSON at the end of the pipeline to return the response.
from `Simple Filter XML Post <https://github.com/edgexfoundry/app-functions-sdk-go/tree/master/examples/simple-filter-xml-post>`_
.. code::
edgexSdk.SetFunctionsPipeline(
edgexSdk.DeviceNameFilter(deviceNames),
edgexSdk.XMLTransform(),
edgexSdk.HTTPPostXML("<Your endpoint goes here>"),
)
Message Bus Trigger
===================
A message bus trigger will execute the pipeline every time data is received off of the configured topic.
Type and Topic Configuration
----------------------------
Here's an example:
.. code::
Type="messagebus"
SubscribeTopic="events"
PublishTopic=""
The Type= is set to "messagebus". EdgeX Core Data is publishing data to the events topic. So to receive data from core data, you can set your SubscribeTopic= either to "" or "events". You may also designate a PublishTopic= if you wish to publish data back to the message bus. edgexcontext.Complete([]byte outputData) - Will send data back to back to the message bus with the topic specified in the PublishTopic= property
Message bus connection configuration
------------------------------------
The other piece of configuration required are the connection settings:
.. code::
[MessageBus]
Type = 'zero' #specifies of message bus (i.e zero for ZMQ)
[MessageBus.PublishHost]
Host = '*'
Port = 5564
Protocol = 'tcp'
[MessageBus.SubscribeHost]
Host = 'localhost'
Port = 5563
Protocol = 'tcp'
By default, EdgeX Core Data publishes data to the events topic on port 5563. The publish host is used if publishing data back to the message bus.
**Important Note:** Publish Host MUST be different for every topic you wish to publish to since the SDK will bind to the specific port. 5563 for example cannot be used to publish since EdgeX Core Data has bound to that port. Similarly, you cannot have two separate instances of the app functions SDK running publishing to the same port.
In the main() function, note the call to MQTTSend at the end of the pipeline to return the response.
from `Simple Filter XML MQTT <https://github.com/edgexfoundry/app-functions-sdk-go/tree/master/examples/simple-filter-xml-mqtt>`_
.. code::
edgexSdk.SetFunctionsPipeline(
edgexSdk.DeviceNameFilter(deviceNames),
edgexSdk.XMLTransform(),
printXMLToConsole,
edgexSdk.MQTTSend(addressable, "", "", 0, false, false),
)
......@@ -8,8 +8,6 @@ Introduction to the SDK
The EdgeX Foundry Device Service Software Development Kit (SDK) takes the Developer through the step-by-step process to create an EdgeX Foundry Device Service microservice. Then setup the SDK and execute the code to generate the Device Service scaffolding to get you started using EdgeX.
The EdgeX Foundry Software Development Kit (SDK) is written in Java. Other languages will be available in the future from the open source community. If you are creating your own SDK, use this one as an example.
The Device Service SDK supports:
* Synchronous read and write operations
......@@ -30,24 +28,24 @@ Device Service Workflow
Key to diagram
+---------------------+-------------------------------------------------------------------------------------------------------------------------------------------+
| **Colour of Box** | **Descrption** |
| **Colour of Box** | **Descrption** |
+=====================+===========================================================================================================================================+
| Orange | Everything is part of a Base Service. |
| Orange | Everything is part of a Base Service. |
+---------------------+-------------------------------------------------------------------------------------------------------------------------------------------+
| Light Green | Initialization. Gets its own configuration and registers itself. |
| Light Green | Initialization. Gets its own configuration and registers itself. |
+---------------------+-------------------------------------------------------------------------------------------------------------------------------------------+
| Yellow | Update Controller. receives, processes, and publishes the update. |
| Yellow | Update Controller. receives, processes, and publishes the update. |
+---------------------+-------------------------------------------------------------------------------------------------------------------------------------------+
| Dark Blue | UInitializing and setting up of schedules. |
| Dark Blue | UInitializing and setting up of schedules. |
+---------------------+-------------------------------------------------------------------------------------------------------------------------------------------+
| Gray | Scaffolding code to be receivers into Device Service. Processes commands. |
| Gray | Scaffolding code to be receivers into Device Service. Processes commands. |
+---------------------+-------------------------------------------------------------------------------------------------------------------------------------------+
| Purple | **Initializes itself.** Set up in metadata. Registers its Device Service discovery process and registration, and sets up Device Services.|
| Purple | **Initializes itself.** Set up in metadata. Registers its Device Service discovery process and registration, and sets up Device Services.|
| | |
| | **Gets Device Watchers.** When a Device Service first comes up it has its initial set of devices. The Device Watcher waits to receive |
| | information that a new device has occurred.!Then a Device Watcher sends metadata messages out about the new device. |
+---------------------+-------------------------------------------------------------------------------------------------------------------------------------------+
| Dark Green | Send data to Core Data. How to communicate with the devices, and on what schedule, and to receive information back from the devices. |
| Dark Green | Send data to Core Data. How to communicate with the devices, and on what schedule, and to receive information back from the devices. |
+---------------------+-------------------------------------------------------------------------------------------------------------------------------------------+
========================
......@@ -55,9 +53,3 @@ Writing a Device Service
========================
`Writing a new Device Service in Go 〉 <Ch-GettingStartedSDK.html>`_
This diff is collapsed.
......@@ -6,14 +6,14 @@ Provison a Device - Modbus Example
..
For this example we will use the GS1-10P5 Modbus motor profile we have available as reference (https://github.com/edgexfoundry/device-modbus/blob/master/src/main/resources/GS1-10P5.profile.yaml, device reference: Marathon Electric MicroMax motors via PLC (http://www.marathon-motors.com/Inverter-Vector-Duty-C-Face-Footed-TEFC-Micromax-Motor_c333.htm)). I would recommend using a tool like Postman for simplifying interactions with the REST APIs (refer to the "Device and Device Service Setup (aka Device Service Creation and Device Provisioning)" section for further details at `API Demo Walkthrough`_ , all REST content is JSON). Also note that Postman is capable of importing RAML documents for API framing (RAML docs for the EdgeX services may be found in src/test/resources/raml/*.raml or on the wiki). Note that this specific example can be tweaked for use with the other Device Services.
For this example we will use the GS1-10P5 Modbus motor profile we have available as reference `GS1 Profile <https://github.com/edgexfoundry/device-modbus/blob/master/src/main/resources/GS1-10P5.profile.yaml>`_ , device reference: Marathon Electric MicroMax motors via PLC (http://www.marathon-motors.com/Inverter-Vector-Duty-C-Face-Footed-TEFC-Micromax-Motor_c333.htm)). I would recommend using a tool like Postman for simplifying interactions with the REST APIs (refer to the "Device and Device Service Setup (aka Device Service Creation and Device Provisioning)" section for further details at `API Demo Walkthrough`_ , all REST content is JSON). Also note that Postman is capable of importing RAML documents for API framing (RAML docs for the EdgeX services may be found in src/test/resources/raml or on the wiki). Note that this specific example can be tweaked for use with the other Device Services.
1. Upload the device profile above to metadata with a POST to http://localhost:48081/api/v1/deviceprofile/uploadfile and add the file as key "file" to the body
2. Add the addressable containing reachability information for the device with a POST to http://localhost:48081/api/v1/addressable:
a. If IP connected, the body will look something like: { "name": "Motor", "method": "GET", "protocol": "HTTP", "address": "10.0.1.29", "port": 502 }
b. If serially connected, the body will look something like: { "name": "Motor", "method": "GET", "protocol": "OTHER", "address": "/dev/ttyS5,9600,8,1,1", "port": 0 } (address field contains port, baud rate, number of data bits, stop bits, and parity bits in CSV form)
3. Ensure the Modbus device service is running, adjust the service name below to match if necessary or if using other device services
4. Add the device with a POST to http://localhost:48081/api/v1/device, the body will look something like:
4. Add the device with a POST to http://localhost:48081/api/v1/device, the body will look something like:
::
......@@ -27,7 +27,7 @@ For this example we will use the GS1-10P5 Modbus motor profile we have available
},
"labels": [
],
"location": null,
"service": {
......@@ -46,11 +46,3 @@ The addressable name must match/refer to the addressable added in Step 2, the se
..
Further deep dives on the different microservices and layers can be found in our EdgeX Tech Talks series (`EdgeX Tech Talks`_.) where Jim and I cover some of the intricacies of various services. Of particular relevance here is the Metadata Part 2 discussion covering Device Profiles and Device Provisioning.
......@@ -4,7 +4,6 @@ Port = 49982
ConnectRetries = 3
Labels = []
OpenMsg = "device mqtt started"
MaxResultCount = 50000
Timeout = 5000
EnableAsyncReadings = true
AsyncBufferSize = 16
......@@ -58,24 +57,37 @@ Level = "DEBUG"
Profile = "Test.Device.MQTT.Profile"
Description = "MQTT device is created for test purpose"
Labels = [ "MQTT", "test"]
[DeviceList.Addressable]
name = "Gateway address"
Protocol = "TCP"
Address = "0.0.0.0"
Port = 1883
Publisher = "CommandPublisher"
user = "admin"
password = "public"
topic = "CommandTopic"
[DeviceList.Protocols]
[DeviceList.Protocols.mqtt]
Schema = "tcp"
Host = "0.0.0.0"
Port = "1883"
ClientId = "CommandPublisher"
User = "admin"
Password = "public"
Topic = "CommandTopic"
[[DeviceList.AutoEvents]]
Frequency = "20s"
OnChange = false
Resource = "testrandfloat32"
# Pre-define Schedule Configuration
[[Schedules]]
Name = "mqtt-30sec-schedule"
Frequency = "PT30S"
[[ScheduleEvents]]
Name = "readRandnum"
Schedule = "mqtt-30sec-schedule"
[ScheduleEvents.Addressable]
HTTPMethod = "GET"
Path = "/api/v1/device/name/MQTT test device/testrandnum"
# Driver configs
[Driver]
IncomingSchema = "tcp"
IncomingHost = "0.0.0.0"
IncomingPort = "1883"
IncomingUser = "admin"
IncomingPassword = "public"
IncomingQos = "0"
IncomingKeepAlive = "3600"
IncomingClientId = "IncomingDataSubscriber"
IncomingTopic = "DataTopic"
ResponseSchema = "tcp"
ResponseHost = "0.0.0.0"
ResponsePort = "1883"
ResponseUser = "admin"
ResponsePassword = "public"
ResponseQos = "0"
ResponseKeepAlive = "3600"
ResponseClientId = "CommandResponseSubscriber"
ResponseTopic = "ResponseTopic"
......@@ -4,7 +4,6 @@ Port = 49991
ConnectRetries = 3
Labels = []
OpenMsg = "device modbus started"
MaxResultCount = 50000
Timeout = 5000
EnableAsyncReadings = true
AsyncBufferSize = 16
......@@ -58,52 +57,26 @@ Level = "DEBUG"
Profile = "Test.Device.Modbus.Profile"
Description = "This device is a product for monitoring and controlling digital inputs and outputs over a LAN."
labels = [ "Air conditioner","modbus TCP" ]
[DeviceList.Addressable]
name = "Gateway address 1"
Protocol = "TCP"
Address = "0.0.0.0"
Port = 1502
Path = "1"
[DeviceList.Protocols]
[DeviceList.Protocols.modbus-tcp]
Address = "0.0.0.0"
Port = "1502"
UnitID = "1"
[[DeviceList.AutoEvents]]
Frequency = "20s"
OnChange = false
Resource = "HVACValues"
[[DeviceList]]
Name = "Modbus RTU test device"
Profile = "Test.Device.Modbus.Profile"
Description = "This device is a product for monitoring and controlling digital inputs and outputs over a LAN."
labels = [ "Air conditioner","modbus RTU" ]
[DeviceList.Addressable]
name = "Gateway address 2"
Protocol = "RTU"
Address = "/tmp/slave,19200,8,1,0"
Path = "1"
# Pre-define Schedule Configuration
[[Schedules]]
Name = "10sec-schedule"
Frequency = "PT10S"
[[ScheduleEvents]]
Name = "Set HVAC Values"
Parameters = "[{\"OperationMode\": \"Auto\"},{\"FanSpeed\": \"Med\"},{\"Temperature\": \"11.22\"}]"
Schedule = "10sec-schedule"
[ScheduleEvents.Addressable]
HTTPMethod = "Put"
Path = "/api/v1/device/name/Modbus TCP test device/HVACValues"
[[ScheduleEvents]]
Name = "Set the Switch"
Parameters = "[{\"SwitchA\": \"ON\"},{\"SwitchB\": \"ON\"}]"
Schedule = "10sec-schedule"
[ScheduleEvents.Addressable]
HTTPMethod = "Put"
Path = "/api/v1/device/name/Modbus TCP test device/Switch"
[[Schedules]]
Name = "5sec-schedule"
Frequency = "PT5S"
[[ScheduleEvents]]
Name = "Read Values"
Schedule = "5sec-schedule"
[ScheduleEvents.Addressable]
HTTPMethod = "GET"
Path = "/api/v1/device/name/Modbus TCP test device/Values"
[DeviceList.Protocols]
[DeviceList.Protocols.modbus-rtu]
Address = "/tmp/slave"
BaudRate = "19200"
DataBits = "8"
StopBits = "1"
Parity = "N"
UnitID = "1"
......@@ -5,21 +5,24 @@ labels:
- "test"
description: "Test device profile"
deviceResources:
-
name: randnum
description: "device random number"
attributes:
{ name: "randnum" }
- name: randfloat32
description: "device random number with Base64 encoding"
properties:
value:
{ type: "Float32", size: "4", readWrite: "R", defaultValue: "0.00", minimum: "100.00", maximum: "0.00", floatEncoding: "Base64" }
units:
{ type: "String", readWrite: "R", defaultValue: "" }
- name: randfloat64
description: "device random number with e notion"
properties:
value:
{ type: "Float64", size: "4", readWrite: "R", defaultValue: "0.00", minimum: "100.00", maximum: "0.00" }
{ type: "Float64", size: "4", readWrite: "R", defaultValue: "0.00", minimum: "100.00", maximum: "0.00", floatEncoding: "eNotation" }
units:
{ type: "String", readWrite: "R", defaultValue: "" }
-
name: ping
description: "device awake"
attributes:
{ name: "ping" }
properties:
value:
{ type: "String", size: "0", readWrite: "R", defaultValue: "oops" }
......@@ -28,44 +31,53 @@ deviceResources:
-
name: message
description: "device notification message"
attributes:
{ name: "message" }
properties:
value:
{ type: "String", size: "0", readWrite: "W" ,scale: "", offset: "", base: "" }
units:
{ type: "String", readWrite: "R", defaultValue: "" }
resources:
-
name: testrandnum
deviceCommands:
- name: testrandfloat32
get:
- { index: "1", operation: "get", object: "randfloat32", parameter: "randfloat32" }
- name: testrandfloat64
get:
- { index: "1", operation: "get", object: "randnum", parameter: "randnum", property: "value" }
- { index: "1", operation: "get", object: "randfloat64", parameter: "randfloat64" }
-
name: testping
get:
- { index: "1", operation: "get", object: "ping", parameter: "ping", property: "value" }
- { index: "1", operation: "get", object: "ping", parameter: "ping" }
-
name: testmessage
get:
- { index: "1", operation: "get", object: "message", parameter: "message", property: "value" }
- { index: "1", operation: "get", object: "message", parameter: "message" }
set:
- { index: "1", operation: "set", object: "message", parameter: "message", property: "value" }
- { index: "1", operation: "set", object: "message", parameter: "message" }
commands:
-
name: testrandnum
coreCommands:
- name: testrandfloat32
get:
path: "/api/v1/device/{deviceId}/testrandnum"
path: "/api/v1/device/{deviceId}/testrandfloat32"
responses:
-
code: "200"
description: "get the random value"
expectedValues: ["randnum"]
description: "get the random float32 value"
expectedValues: ["randfloat32"]
-
code: "503"
description: "service unavailable"
expectedValues: []
- name: testrandfloat64
get:
path: "/api/v1/device/{deviceId}/testrandfloat64"
responses:
- code: "200"
description: "get the random float64 value"
expectedValues: ["randfloat64"]
- code: "503"
description: "service unavailable"
expectedValues: []
-
name: testping
get:
......@@ -103,4 +115,4 @@ commands:
-
code: "503"
description: "service unavailable"
expectedValues: []
expectedValues: []
Application Functions
=====================
The App Functions Software Development Kit (SDK) is a library that is available
for developers to extract and consume events from core data in the EdgeX
Framework. Currently the only supported language is Golang, with the intention
that community developed and supported SDKs will come in the future for other
languages. It is currently available as a Golang module to remain operating
system (OS) agnostic and to comply with the latest EdgeX guidelines on
dependency management.
The intention of the SDK is to address the scalability concerns of the existing
Client and Export Services as well as provide a flexible solution for
exporting data outside of EdgeX without encumbering the EdgeX development
community itself with trying to support all major cloud providers and export
solutions. For the Edinburgh release cycle, the existing Client and Export
Service remain supported and are still considered the primary way to export data
out of EdgeX. However, it is encouraged for new development efforts adopting
EdgeX that the App Functions SDK be leveraged moving forward with the intention
that by the Fuji release, the SDK will be moved into release status and become
the primary method of consuming data from EdgeX.
With the current export services, developers register their endpoints or MQTT
clients with the provided registration services and as events are consumed from
Core Data, the export service would then relay that information to the
registered endpoints in a sequential fashion. Requiring the individual export
service to rebroadcast data to all registered endpoints overtime creates a
bottleneck and leaves applications with a potential delay in receiving events.
Furthermore, the overhead and complexity of managing all registered endpoints
becomes an added burden to EdgeX services. Finally, the Export services have
also begun to address a bit more than is sustainable in regard to supporting all
the major cloud provider endpoints. Providing an SDK and removing cloud specific
exports is one way to remain agnostic to cloud providers and enables 3rd parties
to support their use of any given cloud solution and eliminates the dependency
on EdgeX to support the ever-changing cloud environment.
Providing an SDK that connects
directly to a message bus by which Core Data events are published eliminates
performance issues as well as allow the developers extra control on what happens
with that data as soon as it is available. Furthermore, it emphasizes
configuration over registration for consuming the data. The application services
can be customized to a client's needs and thereby also removing the need for
client registration.
Standard Functions
------------------
One of the most common use cases for working with data that come from
CoreData is to filter data down to what is relevant for a given application
and to format it. To help facilitate this, four primary functions ported
over from the existing services today are included in the SDK. The first is
the `DeviceNameFilter` function which will remove events that do not match the
specified IDs and will cease execution of the pipeline if no event matches.
The second is the `ValueDescriptorFilter` which exhibits the same behavior
as `DeviceNameFilter` except filtering on Value Descriptor instead of
DeviceID. The third and fourth provided functions in the SDK transform the data
received to either XML or JSON by calling `XMLTransform` or
`JSONTransform`.
Typically, after filtering and transforming the data as needed, exporting is
the last step in a pipeline to ship the data where it needs to go. There are
two primary functions included in the SDK to help facilitate this. The first
is `HTTPPost(string url)` function that will POST the provided data to a
specified endpoint, and the second is an `MQTTPublish()` function that will
publish the provided data to an MQTT Broker as specified in the
configuration.
There are two primary triggers that have been included in the SDK that
initiate the start of the function pipeline. First is via a POST HTTP
Endpoint `/trigger` with the EdgeX event data as the body. Second is the
MessageBus subscription with connection details as specified in the
configuration. See `Appendix A – Message Bus Trigger`_.
Finally, data may be sent back to the message bus or HTTP response by
calling.complete() on the context. If the trigger is HTTP, then it will be
an HTTP Response. If the trigger is MessageBus, then it will be published to
the configured host and topic.
Examples
--------
There are three example applications provided in the app-functions-sdk-go
repository in the /examples directory that attempt to show basic structure of
building an application with the app functions sdk. They also focus on how to
leverage various built in provided functions as mentioned above as well as how
to write your own in the case that the SDK does not provide what is needed.
#.
`Simple Filter XML <https://github.com/edgexfoundry/app-functions-sdk-go/tree/master/examples/simple-filter-xml>`_ -> Demonstrates Filter of data by device ID and
transforming data to XML
#.
`Simple Filter XML Post <https://github.com/edgexfoundry/app-functions-sdk-go/tree/master/examples/simple-filter-xml-post>`_ -> Same example as #1, but result published to HTTP
Endpoint
#.
`Simple Filter XML MQTT <https://github.com/edgexfoundry/app-functions-sdk-go/tree/master/examples/simple-filter-xml-mqtt>`_ -> Same example as #1, but result published to MQTT
Broker
The features in the initial implementation of the App Functions SDK should be
sufficient to provide the foundation for basic filtering and export needs. There
are some functions in the existing export services that are not yet available in
application functions and are intended to be included in a later release. This
includes the Encryption Transformer, the Compressor Transformer, and Valid Event
Check. See `Appendix B – Existing export service functions`_. The primary purpose for leaving this out was to
address core pieces of functionality that would set up the ease of adding
additional functions in the future.
Appendices
----------
Appendix A – Message Bus Trigger
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
A message bus trigger will execute the pipeline everytime data is received off
of the configured topic.
Type and Topic configuration
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Here's an example:
.. code-block:: ini
Type="messagebus"
SubscribeTopic="events"
PublishTopic=""
The Type= is set to "messagebus". EdgeX Core Data is publishing data to
the events topic. So to receive data from core data, you can set
your SubscribeTopic= either to "" or "events". You may also designate
a PublishTopic= if you wish to publish data back to the message
bus. edgexcontext.complete([]byte outputData) - Will send data back to back to
the message bus with the topic specified in the PublishTopic= property
Message bus connection configuration
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The other piece of configuration required are the connection settings:
.. code-block:: ini
[MessageBus]
Type = 'zero' #specifies of message bus (i.e zero for ZMQ)
[MessageBus.PublishHost]
Host = '*'
Port = 5564
Protocol = 'tcp'
[MessageBus.SubscribeHost]
Host = 'localhost'
Port = 5563
Protocol = 'tcp'
By default, EdgeX Core Data publishes data to the events topic on port 5563. The
publish host is used if publishing data back to the message bus.
**Important Note:**\  Publish Host \ **MUST**\  be different for every topic you wish
to publish to since the SDK will bind to the specific port. 5563 for example
cannot be used to publish since EdgeX Core Data has bound to that port.
Similarly, you cannot have two separate instances of the app functions SDK
running publishing to the same port.
Appendix B – Existing export service functions
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
From :doc:`./Ch-Distribution`:
**Compressor Transformer**\ –A transformer component compresses the data string to
be delivered to the clients, for any clients that have requested their data be
compressed either by GZIP or ZIP methods.
**Encryption Transformer**\ –An encryption component encrypts the data to be sent
to the client, using the client provided keys and vectors.
**Valid Event Check**\ –The first component in the pipe and filter, before the
copier (described in the previous section) is a filter that can be optionally
turned on or off by configuration. This filter is a general purpose data
checking filter which assesses the device- or sensor-provided Event, with