Upgrade to DHF 4.3

Prerequisites

Data Hub Framework 4.3 requires the following software:

  • MarkLogic Server:

  • Oracle Java 8 JRE

  • For the QuickStart tool: A modern browser, such as Chrome or Firefox

Upgrade Notes and Procedure

Upgrade Notes for DHF 4.3

The following security roles changed in DHF 4.3.

old roles old default users new roles new default users
hub-admin-role hub-admin-user data-hub-admin-role (no default)
    flow-developer-role flow-developer
data-hub-role data-hub-user flow-operator-role flow-operator

You can use your old custom users instead of the new default users. Remember to assign them to the new roles.

For more information on roles in DHF, see Security.

The notes and steps in this tab are for the following upgrade paths:

  • DHF 4.2.x » 4.3.x
  • DHF 4.1.x » 4.3.x

Procedure

  1. In your build.gradle file, replace all occurrences of your old DHF version number with 4.3.2.

    Example: In the plugins section and the dependencies section,

       plugins {
       id 'net.saliman.properties' version '1.4.6'
       id 'com.marklogic.ml-data-hub' version '4.3.2'
   }
   ...
   dependencies {
     compile 'com.marklogic:marklogic-data-hub:4.3.2'
     compile 'com.marklogic:marklogic-xcc:9.0.6'
   }

  2. At your project’s root folder, run the hubUpdate -i Gradle task.

    ./gradlew hubUpdate -i
    gradlew.bat hubUpdate -i

    Result: A gradle-GENERATED.properties file is created.

  3. Update your gradle.properties file based on the gradle-GENERATED.properties file.

    a. Add the following properties and replace the values accordingly.

       mlDHFVersion=4.3.2
   ...
   mlFlowOperatorRole=flow-operator-role
   mlFlowOperatorUserName=flow-operator
   mlFlowOperatorPassword=your-flow-operator-password
   ...
   mlFlowDeveloperRole=flow-developer-role
   mlFlowDeveloperUserName=flow-developer
   mlFlowDeveloperPassword=your-flow-developer-password
   ...
   mlDataHubAdminRole=data-hub-admin-role

    b. Assign default module permissions to the new roles.

       mlModulePermissions=rest-reader,read,rest-writer,insert,rest-writer,update,rest-extension-user,execute,flow-developer-role,read,flow-developer-role,execute,flow-developer-role,insert,flow-operator-role,read,flow-operator-role,execute

  4. If your custom code refers to the old roles/users, change them to refer to the new roles/users.

  5. At your project’s root folder, run the mlDeploy Gradle task.

    ./gradlew mlDeploy
    gradlew.bat mlDeploy

  6. Edit your gradle.properties file again.

    a. Update mlUsername or mlPassword with a new user assigned to flow-developer-role (to create and deploy flows) or to flow-operator-role (to run flows).

       mlUsername=flow-operator
   mlPassword=your-flow-operator-password

    b. Remove the following properties.

    • mlHubUserRole
    • mlHubUserName
    • mlHubUserPassword
    • mlHubAdminRole
    • mlHubAdminUserName
    • mlHubAdminUserPassword

  7. (Optional) Delete the old roles from MarkLogic Server.

    • hub-admin-role
    • data-hub-role

  8. Run your ingest and harmonize flows.

    If you use MarkLogic Content Pump for your input flows, run MLCP with the -transform_module option as follows:

    -transform_module "/data-hub/4/transforms/mlcp-flow-transform.xqy"
    -transform_module "/data-hub/4/transforms/mlcp-flow-transform.sjs"

The notes and steps in this tab are for the following upgrade paths:

  • DHF 4.0.3 » 4.3.x
  • DHF 4.0.2 » 4.3.x
  • DHF 4.0.1 » 4.3.x

Additional Upgrade Notes

  • The hubUpdate task makes the following changes.

    • Archives existing configuration directories under your-project-root/src/main. (4.0.x is the old DHF version.)

      old directory new archive directory
      hub-internal-config hub-internal-config-4.0.x
      ml-config ml-config-4.0.x

    • Overwrites the existing databases, server directories, and the security directory.

  • Running the hubUpdate task with the -i option (info mode) displays specifically what the task does, including configuration settings that changed.

    Example: A verbose report. 
      Upgrading entity-config dir
  Upgrading hub-internal-config dir
  Processing /your-project-root/hub-internal-config/databases/job-database.json
  Setting "schema-database" to "%%mlStagingSchemasDbName%%"
  Setting "triggers-database" to "%%mlStagingTriggersDbName%%"
  Adding path range indexes to job-database.json
  Writing /your-project-root/hub-internal-config/databases/job-database.json to /your-project-root/src/main/hub-internal-config/databases/job-database.json
  Processing /your-project-root/hub-internal-config/databases/final-database.json
  Setting "schema-database" to "%%mlFinalSchemasDbName%%"
  Setting "triggers-database" to "%%mlFinalTriggersDbName%%"
  Writing /your-project-root/hub-internal-config/databases/final-database.json to /your-project-root/src/main/ml-config/databases/final-database.json
  Processing /your-project-root/hub-internal-config/databases/staging-database.json
  Setting "schema-database" to "%%mlStagingSchemasDbName%%"
  Setting "triggers-database" to "%%mlStagingTriggersDbName%%"
  Writing /your-project-root/hub-internal-config/databases/staging-database.json to /your-project-root/src/main/hub-internal-config/databases/staging-database.json
  Writing /your-project-root/hub-internal-config/databases/modules-database.json to /your-project-root/src/main/ml-config/databases/modules-database.json
  Processing /your-project-root/hub-internal-config/servers/job-server.json
  Setting "url-rewriter" to "/data-hub/4/tracing/tracing-rewriter.xml"
  Writing /your-project-root/hub-internal-config/servers/job-server.json to /your-project-root/src/main/hub-internal-config/servers/job-server.json
  Writing /your-project-root/hub-internal-config/servers/final-server.json to /your-project-root/src/main/ml-config/servers/final-server.json
  Processing /your-project-root/hub-internal-config/servers/staging-server.json
  Setting "url-rewriter" to "/data-hub/4/rest-api/rewriter.xml"
  Setting "error-handler" to "/data-hub/4/rest-api/error-handler.xqy"
  Writing /your-project-root/hub-internal-config/servers/staging-server.json to /your-project-root/src/main/hub-internal-config/servers/staging-server.json
  Upgrading user-config dir

Procedure

  1. In your build.gradle file, replace all occurrences of your old DHF version number with 4.3.2.

    Example: In the plugins section and the dependencies section,

       plugins {
       id 'net.saliman.properties' version '1.4.6'
       id 'com.marklogic.ml-data-hub' version '4.3.2'
   }
   ...
   dependencies {
     compile 'com.marklogic:marklogic-data-hub:4.3.2'
     compile 'com.marklogic:marklogic-xcc:9.0.6'
   }

  2. At your project’s root folder, run the hubUpdate -i Gradle task.

    ./gradlew hubUpdate -i
    gradlew.bat hubUpdate -i

    Result: A gradle-GENERATED.properties file is created.

  3. Update your gradle.properties file based on the gradle-GENERATED.properties file.

    a. Add the following properties and replace the values accordingly.

       mlDHFVersion=4.3.2
   ...
   mlFlowOperatorRole=flow-operator-role
   mlFlowOperatorUserName=flow-operator
   mlFlowOperatorPassword=your-flow-operator-password
   ...
   mlFlowDeveloperRole=flow-developer-role
   mlFlowDeveloperUserName=flow-developer
   mlFlowDeveloperPassword=your-flow-developer-password
   ...
   mlDataHubAdminRole=data-hub-admin-role

    b. Assign default module permissions to the new roles.

       mlModulePermissions=rest-reader,read,rest-writer,insert,rest-writer,update,rest-extension-user,execute,flow-developer-role,read,flow-developer-role,execute,flow-developer-role,insert,flow-operator-role,read,flow-operator-role,execute

  4. If your custom code refers to the old roles/users, change them to refer to the new roles/users.

  5. In your-project-root/src/main, copy any custom database/server configurations from the archived configuration files to the new ones. (4.0.x is the old DHF version.)

    copy from files in paste to files in
    hub-internal-config-4.0.x hub-internal-config
    ml-config-4.0.x ml-config

  6. At your project’s root folder, run the mlDeploy Gradle task.

    ./gradlew mlDeploy
    gradlew.bat mlDeploy

  7. Edit your gradle.properties file again.

    a. Update mlUsername or mlPassword with a new user assigned to flow-developer-role (to create and deploy flows) or to flow-operator-role (to run flows).

       mlUsername=flow-operator
   mlPassword=your-flow-operator-password

    b. Remove the following properties.

    • mlHubUserRole
    • mlHubUserName
    • mlHubUserPassword
    • mlHubAdminRole
    • mlHubAdminUserName
    • mlHubAdminUserPassword

  8. (Optional) Delete the old roles from MarkLogic Server.

    • hub-admin-role
    • data-hub-role

  9. Run your ingest and harmonize flows.

    If you use MarkLogic Content Pump for your input flows, run MLCP with the -transform_module option as follows:

    -transform_module "/data-hub/4/transforms/mlcp-flow-transform.xqy"
    -transform_module "/data-hub/4/transforms/mlcp-flow-transform.sjs"

The notes and steps in this tab are for the following upgrade paths:

  • DHF 4.0.0 » 4.3.x

Additional Upgrade Notes

  • DHF 4.0.0 is unique among DHF versions because it has two modules databases: one for the final app server/database and one for the staging app server/database. All other DHF versions before and after 4.0.0 have only one modules database. When upgrading, those databases must be commented out in gradle.properties.

  • The hubUpdate task makes the following changes.

    • Archives existing configuration directories under your-project-root/src/main.

      old directory new archive directory
      hub-internal-config hub-internal-config-4.0.0
      ml-config ml-config-4.0.0

    • Overwrites the existing databases, server directories, and the security directory.

  • Running the hubUpdate task with the -i option (info mode) displays specifically what the task does, including configuration settings that changed.

    Example: A verbose report. 
      Upgrading entity-config dir
  Upgrading hub-internal-config dir
  Processing /your-project-root/hub-internal-config/databases/job-database.json
  Setting "schema-database" to "%%mlStagingSchemasDbName%%"
  Setting "triggers-database" to "%%mlStagingTriggersDbName%%"
  Adding path range indexes to job-database.json
  Writing /your-project-root/hub-internal-config/databases/job-database.json to /your-project-root/src/main/hub-internal-config/databases/job-database.json
  Processing /your-project-root/hub-internal-config/databases/final-database.json
  Setting "schema-database" to "%%mlFinalSchemasDbName%%"
  Setting "triggers-database" to "%%mlFinalTriggersDbName%%"
  Writing /your-project-root/hub-internal-config/databases/final-database.json to /your-project-root/src/main/ml-config/databases/final-database.json
  Processing /your-project-root/hub-internal-config/databases/staging-database.json
  Setting "schema-database" to "%%mlStagingSchemasDbName%%"
  Setting "triggers-database" to "%%mlStagingTriggersDbName%%"
  Writing /your-project-root/hub-internal-config/databases/staging-database.json to /your-project-root/src/main/hub-internal-config/databases/staging-database.json
  Writing /your-project-root/hub-internal-config/databases/modules-database.json to /your-project-root/src/main/ml-config/databases/modules-database.json
  Processing /your-project-root/hub-internal-config/servers/job-server.json
  Setting "url-rewriter" to "/data-hub/4/tracing/tracing-rewriter.xml"
  Writing /your-project-root/hub-internal-config/servers/job-server.json to /your-project-root/src/main/hub-internal-config/servers/job-server.json
  Writing /your-project-root/hub-internal-config/servers/final-server.json to /your-project-root/src/main/ml-config/servers/final-server.json
  Processing /your-project-root/hub-internal-config/servers/staging-server.json
  Setting "url-rewriter" to "/data-hub/4/rest-api/rewriter.xml"
  Setting "error-handler" to "/data-hub/4/rest-api/error-handler.xqy"
  Writing /your-project-root/hub-internal-config/servers/staging-server.json to /your-project-root/src/main/hub-internal-config/servers/staging-server.json
  Upgrading user-config dir

Procedure

  1. In your build.gradle file, replace all occurrences of your old DHF version number with 4.3.2.

    Example: In the plugins section and the dependencies section,

       plugins {
       id 'net.saliman.properties' version '1.4.6'
       id 'com.marklogic.ml-data-hub' version '4.3.2'
   }
   ...
   dependencies {
     compile 'com.marklogic:marklogic-data-hub:4.3.2'
     compile 'com.marklogic:marklogic-xcc:9.0.6'
   }

  2. At your project’s root folder, run the hubUpdate -i Gradle task.

    ./gradlew hubUpdate -i
    gradlew.bat hubUpdate -i

    Result: A gradle-GENERATED.properties file is created.

  3. In your-project-root/src/main, copy any custom database/server configurations from the archived configuration files to the new ones.

    copy from files in paste to files in
    hub-internal-config-4.0.0 hub-internal-config
    ml-config-4.0.0 ml-config

  4. Update your gradle.properties file based on the gradle-GENERATED.properties file.

    a. Add the following properties and replace the values accordingly.

       mlDHFVersion=4.3.2
   ...
   mlFlowOperatorRole=flow-operator-role
   mlFlowOperatorUserName=flow-operator
   mlFlowOperatorPassword=your-flow-operator-password
   ...
   mlFlowDeveloperRole=flow-developer-role
   mlFlowDeveloperUserName=flow-developer
   mlFlowDeveloperPassword=your-flow-developer-password
   ...
   mlDataHubAdminRole=data-hub-admin-role
   ...
   mlModulesDbName=data-hub-MODULES
   mlModulesForestsPerHost=1

    b. Assign default module permissions to the new roles.

       mlModulePermissions=rest-reader,read,rest-writer,insert,rest-writer,update,rest-extension-user,execute,flow-developer-role,read,flow-developer-role,execute,flow-developer-role,insert,flow-operator-role,read,flow-operator-role,execute

    c. Remove the following properties.

    • mlStagingModulesDbName
    • mlStagingModulesForestsPerHost
    • mlStagingModulePermissions
    • mlFinalModulesDbName
    • mlFinalModulesForestsPerHost
    • mlFinalModulePermissions

  5. If your custom code refers to the old roles/users, change them to refer to the new roles/users.

  6. At your project’s root folder, run the mlDeploy Gradle task.

    ./gradlew mlDeploy
    gradlew.bat mlDeploy

  7. Edit your gradle.properties file again.

    a. Update mlUsername or mlPassword with a new user assigned to flow-developer-role (to create and deploy flows) or to flow-operator-role (to run flows).

       mlUsername=flow-operator
   mlPassword=your-flow-operator-password

    b. Remove the following properties.

    • mlHubUserRole
    • mlHubUserName
    • mlHubUserPassword
    • mlHubAdminRole
    • mlHubAdminUserName
    • mlHubAdminUserPassword

  8. (Optional) Delete the old roles from MarkLogic Server.

    • hub-admin-role
    • data-hub-role

  9. Run your ingest and harmonize flows.

    If you use MarkLogic Content Pump for your input flows, run MLCP with the -transform_module option as follows:

    -transform_module "/data-hub/4/transforms/mlcp-flow-transform.xqy"
    -transform_module "/data-hub/4/transforms/mlcp-flow-transform.sjs"

Remarks

After running mlUndeploy, delete the following obsolete resources:

  • data-hub-staging-MODULES database and forest
  • data-hub-final-MODULES database and forest

The notes and steps in this tab are for the following upgrade paths:

  • DHF 3.0.0 » 4.3.x
  • DHF 2.0.6 » 4.3.x
  • DHF 2.0.5 » 4.3.x
  • DHF 2.0.4 » 4.3.x

Additional Upgrade Notes

  • The hubUpdate Gradle task makes the following changes.

    • Renames old configuration directories under your project root.

      old directory new directory
      hub-internal-config hub-internal-config.old
      user-config user-config.old
      entity-config entity-config.old

    • Creates the new project directory structure (your-project-root/src/main and its subdirectories) and new files.

    • Copies some settings from the old configuration files to the new ones.

    • Updates all flows to use updated imports. See the notes to upgrade to 4.0.x.

  • Running the hubUpdate task with the -i option (info mode) displays specifically what the task does, including configuration settings that changed.

    Example: A verbose report. 
      Upgrading entity-config dir
  Upgrading hub-internal-config dir
  Processing /your-project-root/hub-internal-config/databases/job-database.json
  Setting "schema-database" to "%%mlStagingSchemasDbName%%"
  Setting "triggers-database" to "%%mlStagingTriggersDbName%%"
  Adding path range indexes to job-database.json
  Writing /your-project-root/hub-internal-config/databases/job-database.json to /your-project-root/src/main/hub-internal-config/databases/job-database.json
  Processing /your-project-root/hub-internal-config/databases/final-database.json
  Setting "schema-database" to "%%mlFinalSchemasDbName%%"
  Setting "triggers-database" to "%%mlFinalTriggersDbName%%"
  Writing /your-project-root/hub-internal-config/databases/final-database.json to /your-project-root/src/main/ml-config/databases/final-database.json
  Processing /your-project-root/hub-internal-config/databases/staging-database.json
  Setting "schema-database" to "%%mlStagingSchemasDbName%%"
  Setting "triggers-database" to "%%mlStagingTriggersDbName%%"
  Writing /your-project-root/hub-internal-config/databases/staging-database.json to /your-project-root/src/main/hub-internal-config/databases/staging-database.json
  Writing /your-project-root/hub-internal-config/databases/modules-database.json to /your-project-root/src/main/ml-config/databases/modules-database.json
  Processing /your-project-root/hub-internal-config/servers/job-server.json
  Setting "url-rewriter" to "/data-hub/4/tracing/tracing-rewriter.xml"
  Writing /your-project-root/hub-internal-config/servers/job-server.json to /your-project-root/src/main/hub-internal-config/servers/job-server.json
  Writing /your-project-root/hub-internal-config/servers/final-server.json to /your-project-root/src/main/ml-config/servers/final-server.json
  Processing /your-project-root/hub-internal-config/servers/staging-server.json
  Setting "url-rewriter" to "/data-hub/4/rest-api/rewriter.xml"
  Setting "error-handler" to "/data-hub/4/rest-api/error-handler.xqy"
  Writing /your-project-root/hub-internal-config/servers/staging-server.json to /your-project-root/src/main/hub-internal-config/servers/staging-server.json
  Upgrading user-config dir

  • If custom configurations (i.e., from user-config) are missing, you must manually copy them to ml-config.

  • Because DHF 3.0.0 and DHF 2.0.4+ had only a single SCHEMAS database and a single TRIGGERS database, you must decide whether to use those existing databases as the staging databases or as the final databases in DHF 4.x. The settings in gradle.properties (and possibly other configurations) depend on your decision.

    DHF 3.0.0 and 2.0.4+ DHF 4.x
    data-hub-SCHEMAS database data-hub-staging-SCHEMAS database
    data-hub-final-SCHEMAS database
    data-hub-TRIGGERS database data-hub-staging-TRIGGERS database
    data-hub-final-TRIGGERS database

Procedure

  1. In your build.gradle file, replace all occurrences of your old DHF version number with 4.3.2.

    Example: In the plugins section and the dependencies section,

       plugins {
       id 'net.saliman.properties' version '1.4.6'
       id 'com.marklogic.ml-data-hub' version '4.3.2'
   }
   ...
   dependencies {
     compile 'com.marklogic:marklogic-data-hub:4.3.2'
     compile 'com.marklogic:marklogic-xcc:9.0.6'
   }

  2. At your project’s root folder, run the hubUpdate -i Gradle task.

    ./gradlew hubUpdate -i
    gradlew.bat hubUpdate -i

    Result: A gradle-GENERATED.properties file is created.

  3. Update your gradle.properties file based on the gradle-GENERATED.properties file.

    a. Add the following properties and replace the values accordingly.

       mlDHFVersion=4.3.2
   ...
   mlFlowOperatorRole=flow-operator-role
   mlFlowOperatorUserName=flow-operator
   mlFlowOperatorPassword=your-flow-operator-password
   ...
   mlFlowDeveloperRole=flow-developer-role
   mlFlowDeveloperUserName=flow-developer
   mlFlowDeveloperPassword=your-flow-developer-password
   ...
   mlDataHubAdminRole=data-hub-admin-role
   ...
   mlStagingTriggersDbName=data-hub-staging-TRIGGERS
   mlStagingTriggersForestsPerHost=1
   mlStagingSchemasDbName=data-hub-staging-SCHEMAS
   mlStagingSchemasForestsPerHost=1
   ...
   mlFinalTriggersDbName=data-hub-final-TRIGGERS
   mlFinalTriggersForestsPerHost=1
   mlFinalSchemasDbName=data-hub-final-SCHEMAS
   mlFinalSchemasForestsPerHost=1

    b. Assign default module permissions to the new roles.

       mlModulePermissions=rest-reader,read,rest-writer,insert,rest-writer,update,rest-extension-user,execute,flow-developer-role,read,flow-developer-role,execute,flow-developer-role,insert,flow-operator-role,read,flow-operator-role,execute

    c. Remove the following properties.

    • data-hub-TRACING server
    • data-hub-TRACING database
    • data-hub-TRIGGERS database
    • data-hub-SCHEMAS database

  4. If your custom code refers to the old roles/users, change them to refer to the new roles/users.

  5. At your project’s root folder, run the mlDeploy Gradle task.

    ./gradlew mlDeploy
    gradlew.bat mlDeploy

  6. Edit your gradle.properties file again.

    a. Update mlUsername or mlPassword with a new user assigned to flow-developer-role (to create and deploy flows) or to flow-operator-role (to run flows).

       mlUsername=flow-operator
   mlPassword=your-flow-operator-password

    b. Remove the following properties.

    • mlHubUserRole
    • mlHubUserName
    • mlHubUserPassword
    • mlHubAdminRole
    • mlHubAdminUserName
    • mlHubAdminUserPassword

  7. (Optional) Delete the old roles from MarkLogic Server.

    • hub-admin-role
    • data-hub-role

  8. Run your ingest and harmonize flows.

    If you use MarkLogic Content Pump for your input flows, run MLCP with the -transform_module option as follows:

    -transform_module "/data-hub/4/transforms/mlcp-flow-transform.xqy"
    -transform_module "/data-hub/4/transforms/mlcp-flow-transform.sjs"

Remarks

Before running mlUndeploy, delete the resources associated with the properties that were removed from the gradle.properties file:

  • data-hub-TRACING server
  • data-hub-TRACING database
  • data-hub-TRIGGERS database
  • data-hub-SCHEMAS database

The notes and steps in this tab are for the following upgrade paths:

  • DHF 2.0.3 and earlier versions » 4.3.x

Procedure

  1. Upgrade to DHF 2.0.6.
  2. Follow the steps to upgrade from 3.0.0 or 2.0.4+ to DHF 4.3.x.

See Also