Commit b160daec authored by Trevor Conn's avatar Trevor Conn Committed by GitHub

Patch docker-compose timing issues in export/rules-engine (#1550)

Fix #1544

The primary issue is observed when one runs "docker-compose up" after
pruning local docker volumes.

The following problematic symptoms have been observed. The requisite
code changes in this PR are enumerated below. I hesitate to properly
call this a "fix" because I think more analysis is warranted for a long
term solution. However for the purpose of stabilizing the release, this
should suffice.

- Observed that population of configuration data in Consul and
  subsequent creation of DBs/Collections and credentials in Mongo were
  delayed when creating volumes for the first time. This lead to timing
  issues in the following areas
- I observed conditions where the Config-Seed was populating Consul for
  a given service at the same time that that service's configuration was
  being read into Consul by the config-seed. Thus services would come
  up and read their configuration from Consul when that config was only
  partially populated. In Delhi this would not be a problem because the
  listenForConfigChanges() function would listen for a change anywhere
  in the service's config in Consul. This would cause the entirety of
  the ConfigurationStruct to be populated when the last key was written.
  In Edinburgh, we only listen for changes to the "Writable" section,
  which is written first.
  -- This is why initializeConfiguration() has been modified to look at
     the last value of the service's configuration within Consul to see
     if it has been populated. If it is not, then we throw an error --
     forcing a retry one second later. If it is populated, the whole
     config has been written to Consul and we can proceed.
- All services write logging information when they are bootstrapping.
  If the service is configured to log remotely, those calls go to
  support-logging. Because of the observed latency (up to 5 seconds) in
  populating configuration and Mongo with new volumes, any service trying
  to send its startup messages to support-logging would throw an error.
  This lead to many, mnay errors in the starup process. Once support-
  logging came up, the remote logging would work. Understand that every
  service must receive its configuration information from Consul and,
  where applicable, establish DB connectivity before it activates its
  API handlers. If a service calls support-logging (or any other service)
  before the API is enabled, the caller will receive a "connection re-
  fused".
  -- This was also mitigated by the code change above.
- In the case where a service is delayed in its startup, it will not
  register its endpoint with Consul. Export-client does not require a
  database connection and so it will come up before export-distro. It
  was observed that the rules-engine registers with export-client in
  order to receive events. As a result of that, export-client attempts
  to POST a notification to export-distro. Because export-distro had not
  registered itself with Consul at the time export-client was being
  configured, the endpoint information would be blank, leading to an
  error like this (some fields omitted):
    level=ERROR app=edgex-export-client source=registration.go:310
    msg="error from distro: Put http://:0/api/v1/notify/registrations:
         dial tcp :0: connect: connection refused"
  Notice the the host and port.
  -- The code change to address this can be found in
     internal/pkg/startup/endpoint.go
  -- This only works due to the non-HA mode in which we deploy from
     docker-compose. Right now, what's in the docker configuration is
     always the same as what is (or will be) in Consul.
  -- Note also that there is a circular dependency between distro/client
     w/r/t how they call each other. Because the result of a notification
     sent to distro in the above case is that distro simply calls back
     to export-client.
Signed-off-by: default avatarTrevor Conn <trevor_conn@dell.com>
parent da634528
......@@ -6,7 +6,7 @@ require (
github.com/OneOfOne/xxhash v1.2.5
github.com/eclipse/paho.mqtt.golang v1.1.1
github.com/edgexfoundry/go-mod-core-contracts v0.1.0
github.com/edgexfoundry/go-mod-messaging v0.1.5
github.com/edgexfoundry/go-mod-messaging v0.1.8
github.com/edgexfoundry/go-mod-registry v0.1.0
github.com/globalsign/mgo v0.0.0-20181015135952-eeefdecb41b8
github.com/go-kit/kit v0.8.0
......
......@@ -138,7 +138,11 @@ func initializeConfiguration(useRegistry bool, useProfile string) (*Configuratio
configuration = actual
// Check that information was successfully read from Registry
if configuration.Service.Port == 0 {
client, ok := configuration.Clients["Logging"]
if !ok {
return nil, errors.New("error reading configuration from Registry")
}
if client.Port == 0 {
return nil, errors.New("error reading configuration from Registry")
}
}
......
......@@ -199,7 +199,7 @@ func initializeConfiguration(useRegistry bool, useProfile string) (*Configuratio
configuration = actual
// Check that information was successfully read from Registry
if configuration.Service.Port == 0 {
if configuration.MessageQueue.Topic == "" {
return nil, errors.New("error reading configuration from Registry")
}
}
......
......@@ -139,7 +139,7 @@ func initializeConfiguration(useRegistry bool, useProfile string) (*Configuratio
configuration = actual
// Check that information was successfully read from Registry
if configuration.Service.Port == 0 {
if configuration.Notifications.Label == "" {
return nil, errors.New("error reading configuration from Registry")
}
}
......
......@@ -177,7 +177,11 @@ func initializeConfiguration(useRegistry bool, useProfile string) (*Configuratio
configuration = actual
// Check that information was successfully read from Registry
if configuration.Service.Port == 0 {
client, ok := configuration.Databases["Primary"]
if !ok {
return nil, errors.New("error reading configuration from Registry")
}
if client.Type == "" {
return nil, errors.New("error reading configuration from Registry")
}
}
......
......@@ -155,6 +155,7 @@ func addReg(w http.ResponseWriter, r *http.Request) {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
LoggingClient.Info(fmt.Sprintf("Registration added %s", reg.Name))
notifyUpdatedRegistrations(models.NotifyUpdate{Name: reg.Name,
Operation: "add"})
......
......@@ -212,7 +212,7 @@ func initializeConfiguration(useRegistry bool, useProfile string) (*Configuratio
configuration = actual
// Check that information was successfully read from Registry
if configuration.Service.Port == 0 {
if configuration.AnalyticsQueue.Type == "" {
return nil, errors.New("error reading configuration from Registry")
}
}
......
......@@ -45,7 +45,14 @@ func (e Endpoint) Monitor(params types.EndpointParams, ch chan string) {
if err != nil {
fmt.Fprintln(os.Stdout, err.Error())
}
url := fmt.Sprintf("http://%s:%v%s", endpoint.Host, endpoint.Port, params.Path)
var url string
if endpoint.Host != "" {
url = fmt.Sprintf("http://%s:%v%s", endpoint.Host, endpoint.Port, params.Path)
} else {
url = params.Url
}
ch <- url
time.Sleep(time.Millisecond * time.Duration(params.Interval))
}
......
......@@ -126,7 +126,11 @@ func initializeConfiguration(useRegistry bool, useProfile string) (*Configuratio
configuration = actual
// Check that information was successfully read from Registry
if configuration.Service.Port == 0 {
client, ok := configuration.Databases["Primary"]
if !ok {
return nil, errors.New("error reading configuration from Registry")
}
if client.Type == "" {
return nil, errors.New("error reading configuration from Registry")
}
}
......
......@@ -177,7 +177,7 @@ func initializeConfiguration(useRegistry bool, useProfile string) (*Configuratio
configuration = actual
// Check that information was successfully read from Registry
if configuration.Service.Port == 0 {
if configuration.Smtp.Subject == "" {
return nil, errors.New("error reading configuration from Registry")
}
}
......
......@@ -163,7 +163,11 @@ func initializeConfiguration(useRegistry bool, useProfile string) (*Configuratio
configuration = actual
// Check that information was successfully read from Registry
if configuration.Service.Port == 0 {
client, ok := configuration.IntervalActions["ScrubAged"]
if !ok {
return nil, errors.New("error reading configuration from Registry")
}
if client.Interval == "" {
return nil, errors.New("error reading configuration from Registry")
}
}
......
......@@ -9,4 +9,4 @@
package edgex
// Global version for edgex-go
var Version string = "1.0.0"
var Version string = "1.0.1"
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment