2021-11-17

Release Notes / Changelog

Summary

Hodgepodge of k8s podspec changes for Java-based services, and minor enhancements across many of the RCKMS services in support of parallel production testing and validation.

To support garbage collector (GC) logging in Java-based services, Kubernetes Pod specifications for most of the Java-based services will need to be amended to support a new shared volume mount, a sidecar container running Busybox (to tail logs into a log capture daemon), and updated Java Options environment variables to enable this output.

While these amendments are similar, there are key differences for some of these services, by which we would recommend reviewing prior to applying to environments.

DSS

• RCKMSDEV-452: validate Glassfish/Tomcat configurations are similar to existing RCKMS production values (as each container will have it's own Glassfish/Tomcat config).

To enable GC logging, please update the Kubernetes pod specification for DSS:

In the primary DSS container, add a volume mount:

volume_mount {
  name       = "java-diag"
  mount_path = "/hln/diagnostics"
}

In the primary DSS container, modify (or add) the following environment variable, keyed JAVA_OPTS with content: -Xms16g -Xmx16g -XX:+UseG1GC -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:/hln/diagnostics/garbageCollection.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=1 -XX:GCLogFileSize=10m -XX:+UseStringDeduplication

NOTE: the Xms and Xmx values in the preceding environment variable should be less than or equal to 80% of the RAM allocated to each pod. For AIMS Sandbox, Onboard, and PRR, these values can be lower than the recommended values for production (for DSS, we recommend allocating 20GB of RAM to the primary container).

Next, add the following sidecar container specification to DSS:

        # Busybox GC Logger
        container {
          name              = "gc-logger"
          image             = "busybox:latest"
          image_pull_policy = ""
          command = [
            "/bin/sh"
          ]
          args = [
            "-c",
            "tail -F -v /hln/diagnostics/garbageCollection.log"
          ]
          resources {
            limits = {
              cpu    = "150m"
              memory = "128Mi"
            }
            requests = {
              cpu    = "150m"
              memory = "128Mi"
            }
          }
          security_context {
            allow_privilege_escalation = false
            # non-root user
            run_as_user     = 1000
            run_as_non_root = true
          }
          volume_mount {
            name       = "java-diag"
            mount_path = "/hln/diagnostics"
          }
        }

Finally, add the following volume definition to the DSS pod specification:

volume {
  name = "java-diag"
  empty_dir {
    medium = ""
  }
}

DSUS

• RCKMSDEV-530: Add MongoDB connectivity check to k8s probe endpoints.

MTS

• RCKMSDEV-452: validate Glassfish/Tomcat configurations are similar to existing RCKMS production values (as each container will have it's own Glassfish/Tomcat config).

To improve reliability of the MTS container with production datasets, please add the following environment variable key value pairs to the MTS container specification:

• TCP_WRITE_TIMEOUT = 180000

To enable GC logging, please update the Kubernetes pod specification for MTS:

In the primary MTS container, add a volume mount:

volume_mount {
  name       = "java-diag"
  mount_path = "/hln/diagnostics"
}

In the primary MTS container, modify (or add) the following environment variable, keyed JAVA_OPTS with content -Xms8g -Xmx8g -XX:+UseG1GC -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:/hln/diagnostics/garbageCollection.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=1 -XX:GCLogFileSize=10m -XX:+UseStringDeduplication

NOTE: the Xms and Xmx values in the preceding environment variable should be less than or equal to 80% of the RAM allocated to each pod. For AIMS Sandbox, Onboard, and PRR, these values can be lower than the recommended values for production (for MTS, we recommend allocating 10GB of RAM to the primary container).

Next, add the following sidecar container specification to MTS:

        # Busybox GC Logger
        container {
          name              = "gc-logger"
          image             = "busybox:latest"
          image_pull_policy = ""
          command = [
            "/bin/sh"
          ]
          args = [
            "-c",
            "tail -F -v /hln/diagnostics/garbageCollection.log"
          ]
          resources {
            limits = {
              cpu    = "150m"
              memory = "128Mi"
            }
            requests = {
              cpu    = "150m"
              memory = "128Mi"
            }
          }
          security_context {
            allow_privilege_escalation = false
            # non-root user
            run_as_user     = 1000
            run_as_non_root = true
          }
          volume_mount {
            name       = "java-diag"
            mount_path = "/hln/diagnostics"
          }
        }

Finally, add the following volume definition to the MTS pod specification:

volume {
  name = "java-diag"
  empty_dir {
    medium = ""
  }
}

OUS

• RCKMSDEV-530: Add MongoDB connectivity check to k8s probe endpoints.

SS

• RCKMSDEV-531: Thinning of the logs.

• RCKMSDEV-525: add operator diagnostic endpoint (accessible at GET /__/diagnostics) to validate SS configuration.

To enable support for this new diagnostic endpoint, please add the following environment variable key value pairs to the SS container specification:

• SERVICE_BASE_DSS, which should be the fully-qualified URL to DSS within the service mesh (e.g. http://dss.svc.cluster.local)

• SERVICE_BASE_VCS, which should be the fully-qualified URL to VCS within the service mesh (e.g. http://vcs.svc.cluster.local)

SSCS

• RCKMSDEV-344: add support for serviceResponseTime configuration variable.

To enable GC logging, please update the Kubernetes pod specification for SSCS:

In the primary SSCS container, add a volume mount:

volume_mount {
  name       = "java-diag"
  mount_path = "/hln/diagnostics"
}

In the primary SSCS container, modify (or add) the following environment variable, keyed JAVA_TOOL_OPTIONS with content -Xmx4g -Xms4g -XX:+UseG1GC -Xlog:gc*,gc+phases=debug:file=/hln/diagnostics/garbageCollection.log:uptime,utctime,level,tags,pid,hostname:filesize=10m,filecount=1 -XX:+UseStringDeduplication

NOTE: the Xms and Xmx values in the preceding environment variable should be less than or equal to 80% of the RAM allocated to each pod. For AIMS Sandbox, Onboard, and PRR, these values can be lower than the recommended values for production (for SSCS, we recommend allocating 5GB of RAM to the primary container).

Next, add the following sidecar container specification to SSCS:

        # Busybox GC Logger
        container {
          name              = "gc-logger"
          image             = "busybox:latest"
          image_pull_policy = ""
          command = [
            "/bin/sh"
          ]
          args = [
            "-c",
            "tail -F -v /hln/diagnostics/garbageCollection.log"
          ]
          resources {
            limits = {
              cpu    = "150m"
              memory = "128Mi"
            }
            requests = {
              cpu    = "150m"
              memory = "128Mi"
            }
          }
          security_context {
            allow_privilege_escalation = false
            # non-root user
            run_as_user     = 1000
            run_as_non_root = true
          }
          volume_mount {
            name       = "java-diag"
            mount_path = "/hln/diagnostics"
          }
        }

Finally, add the following volume definition to the SSCS pod specification:

volume {
  name = "java-diag"
  empty_dir {
    medium = ""
  }
}

RGS

• RCKMSDEV-526: add diagnostic endpoint to RGS.

• RCKMSDEV-508: Modify the order of Predicates so that embedded Concepts (such as Entity) are searched for at the proper time in Drools aligned with the vMR XPath.

• RCKMSDEV-509: When wrapped by a function, an embedded Predicate Group that follows a Predicate is not properly joining. For example, the below ObservationValues are not joining to the ObservationFocus

• RCKMSDEV-452: validate Glassfish/Tomcat configurations are similar to existing RCKMS production values (as each container will have it's own Glassfish/Tomcat config).

To enable GC logging, please update the Kubernetes pod specification for RGS:

In the primary RGS container, add a volume mount:

volume_mount {
  name       = "java-diag"
  mount_path = "/hln/diagnostics"
}

In the primary RGS container, modify (or add) the following environment variable, keyed JAVA_TOOL_OPTIONS with content -Xmx4g -Xms4g -XX:+UseG1GC -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:/hln/diagnostics/garbageCollection.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=1 -XX:GCLogFileSize=10m -XX:+UseStringDeduplication

NOTE: the Xms and Xmx values in the preceding environment variable should be less than or equal to 80% of the RAM allocated to each pod. For AIMS Sandbox, Onboard, and PRR, these values can be lower than the recommended values for production (for RGS, we recommend allocating 5GB of RAM to the primary container).

Next, add the following sidecar container specification to RGS:

        # Busybox GC Logger
        container {
          name              = "gc-logger"
          image             = "busybox:latest"
          image_pull_policy = ""
          command = [
            "/bin/sh"
          ]
          args = [
            "-c",
            "tail -F -v /hln/diagnostics/garbageCollection.log"
          ]
          resources {
            limits = {
              cpu    = "150m"
              memory = "128Mi"
            }
            requests = {
              cpu    = "150m"
              memory = "128Mi"
            }
          }
          security_context {
            allow_privilege_escalation = false
            # non-root user
            run_as_user     = 1000
            run_as_non_root = true
          }
          volume_mount {
            name       = "java-diag"
            mount_path = "/hln/diagnostics"
          }
        }

Finally, add the following volume definition to the RGS pod specification:

volume {
  name = "java-diag"
  empty_dir {
    medium = ""
  }
}

VCS

• RCKMSDEV-452: validate Glassfish/Tomcat configurations are similar to existing RCKMS production values (as each container will have it's own Glassfish/Tomcat config).

To enable GC logging, please update the Kubernetes pod specification for VCS:

In the primary VCS container, add a volume mount:

volume_mount {
  name       = "java-diag"
  mount_path = "/hln/diagnostics"
}

In the primary VCS container, modify (or add) the following environment variable, keyed JAVA_TOOL_OPTIONS with content -Xmx4g -Xms4g -XX:+UseG1GC -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:/hln/diagnostics/garbageCollection.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=1 -XX:GCLogFileSize=10m -XX:+UseStringDeduplication

NOTE: the Xms and Xmx values in the preceding environment variable should be less than or equal to 80% of the RAM allocated to each pod. For AIMS Sandbox, Onboard, and PRR, these values can be lower than the recommended values for production (for VCS, we recommend allocating 5GB of RAM to the primary container).

Next, add the following sidecar container specification to VCS:

        # Busybox GC Logger
        container {
          name              = "gc-logger"
          image             = "busybox:latest"
          image_pull_policy = ""
          command = [
            "/bin/sh"
          ]
          args = [
            "-c",
            "tail -F -v /hln/diagnostics/garbageCollection.log"
          ]
          resources {
            limits = {
              cpu    = "150m"
              memory = "128Mi"
            }
            requests = {
              cpu    = "150m"
              memory = "128Mi"
            }
          }
          security_context {
            allow_privilege_escalation = false
            # non-root user
            run_as_user     = 1000
            run_as_non_root = true
          }
          volume_mount {
            name       = "java-diag"
            mount_path = "/hln/diagnostics"
          }
        }

Finally, add the following volume definition to the VCS pod specification:

volume {
  name = "java-diag"
  empty_dir {
    medium = ""
  }
}

Bill of Materials

Changes to component SHA1 / Tag values (indicating a release) are marked as emboldened line entries. Components link to their respective documentation, and tags link to the Github repository release for that individual component.

Component	Shortname	SHA1	Tag
cat-rckms	CAT	4b422d3	2.11.1
data-support-update-service	DSUS	ca7d23a	1.6.2
decision-support-service	DSS	9a74c55	2.3.0
dss-preflight-container	DSS-PFC	eb564a1	1.1.1
middle-tier-service	MTS	3cf5306	2.4.0
opencds-update-service	OUS	3525806	1.6.0
rckms-reports-service	RRS	f11c8be	1.1.0
rules-generation-service	RGS	463fd9d	2.0.16
shared-service	SS	88aa151	1.6.0
ss-comparison-service	SSCS	10716c6	1.2.0
vmr-converter-service	VCS	cbe2cce	2.3.0