Project Directory Structure for DHF 4.1.x

A DHF 4.1.x project will have the following directory structure after initialization (through QuickStart or the hubInit Gradle task).

Show complete directory structure.

  your-project-root
  ├─ build.gradle
  ├─ gradle.properties
  ├─ gradle-env1.properties
  ├─ ...
  ├─ gradle-envN.properties
  ├─ gradlew
  ├─ gradlew.bat
  ├─ gradle
  │  └─ wrapper
  │     ├─ gradle-wrapper.jar
  │     └─ gradle-wrapper.properties
  ├─ plugins
  │  ├─ entities
  │  │  ├─ entity1
  │  │  │  ├─ input
  │  │  │  │  ├─ inputflow1
  │  │  │  │  │  ├─ content.(sjs|xqy)
  │  │  │  │  │  ├─ headers.(sjs|xqy)
  │  │  │  │  │  ├─ main.(sjs|xqy)
  │  │  │  │  │  └─ triples.(sjs|xqy)
  │  │  │  │  ├─ ...
  │  │  │  │  └─ inputflowN
  │  │  │  └─ harmonize
  │  │  │     ├─ harmonizeflow1
  │  │  │     │  ├─ collector.(sjs|xqy)
  │  │  │     │  ├─ content.(sjs|xqy)
  │  │  │     │  ├─ headers.(sjs|xqy)
  │  │  │     │  ├─ main.(sjs|xqy)
  │  │  │     │  ├─ triples.(sjs|xqy)
  │  │  │     │  └─ writer.(sjs|xqy)
  │  │  │     ├─ ...
  │  │  │     └─ harmonizeflowN
  │  │  ├─ ...
  │  │  └─ entityN
  │  └─ mappings
  │     ├─ mappingName1
  │     │  ├─ mappingName-0.mapping.json
  │     │  ├─ ...
  │     │  └─ mappingName-N.mapping.json
  │     ├─ ...
  │     └─ mappingNameN
  ├─ src
  │  └─ main
  │     ├─ entity-config
  │     │  ├─ final-entity-options.xml
  │     │  ├─ staging-entity-options.xml
  │     │  └─ databases
  │     │     ├─ final-database.json
  │     │     └─ staging-database.json
  │     ├─ hub-internal-config
  │     │  ├─ databases
  │     │  │  ├─ job-database.json
  │     │  │  ├─ staging-database.json
  │     │  │  ├─ staging-schemas-database.json
  │     │  │  └─ staging-triggers-database.json
  │     │  ├─ schemas
  │     │  ├─ security
  │     │  │  ├─ privileges
  │     │  │  │  ├─ dhf-internal-data-hub.json
  │     │  │  │  ├─ dhf-internal-entities.json
  │     │  │  │  ├─ dhf-internal-mappings.json
  │     │  │  │  └─ dhf-internal-trace-ui.json
  │     │  │  ├─ roles
  │     │  │  │  ├─ data-hub-role.json
  │     │  │  │  └─ hub-admin-role.json
  │     │  │  └─ users
  │     │  │     ├─ data-hub-user.json
  │     │  │     └─ hub-admin-user.json
  │     │  ├─ servers
  │     │  │  ├─ job-server.json
  │     │  │  └─ staging-server.json
  │     │  └─ triggers
  │     ├─ ml-config
  │     │  ├─ databases
  │     │  │  ├─ final-database.json
  │     │  │  ├─ final-schemas-database.json
  │     │  │  ├─ final-triggers-database.json
  │     │  │  └─ modules-database.json
  │     │  ├─ entities.layout.json
  │     │  ├─ security
  │     │  │  ├─ privileges
  │     │  │  ├─ roles
  │     │  │  └─ users
  │     │  ├─ servers
  │     │  │  └─ final-server.json
  │     │  └─ triggers
  │     ├─ ml-modules
  │     └─ ml-schemas
  └─ .tmp

root

  your-project-root
  ├─ build.gradle
  ├─ gradle.properties
  ├─ gradle-env1.properties
  ├─ ...
  ├─ gradle-envN.properties
  ├─ gradlew
  ├─ gradlew.bat
  ├─ gradle
  ├─ plugins
  ├─ src
  └─ .tmp

build.gradle: This file enables you to use Gradle to configure and manage your data hub instance. See the Gradle documentation.
gradle.properties: This properties file defines variables needed by the data hub to install and run properly. Use this file to store values that apply to all instances of your data hub.
gradle-env.properties: DHF determines your project’s various environments (e.g.: dev, qa, prod, local) based on the existence of override files in your hub project. To create a new environment, simply create a new override file with the environment name after the dash. Example: The gradle-local.properties file contains settings that override the variables in gradle.properties for your local environment.
gradlew, gradlew.bat: These are Unix/Linux and Windows executable files that run the Gradle wrapper in the gradle directory.

gradle

  gradle
  └─ wrapper
     ├─ gradle-wrapper.jar
     └─ gradle-wrapper.properties

This directory contains the Gradle wrapper, which is a custom local version of Gradle, so Gradle doesn’t have to be installed separately. The Gradle wrapper is installed when you initialize a new DHF project.

plugins

  plugins
  ├─ entities
  │  ├─ entity1
  │  │  ├─ input
  │  │  │  ├─ inputflow1
  │  │  │  ├─ ...
  │  │  │  └─ inputflowN
  │  │  └─ harmonize
  │  │     ├─ harmonizeflow1
  │  │     ├─ ...
  │  │     └─ harmonizeflowN
  │  ├─ ...
  │  └─ entityN
  └─ mappings
     ├─ mappingName1
     ├─ ...
     └─ mappingNameN

This directory contains project-specific server-side modules that are deployed into MarkLogic.

You can add custom server-side files in this directory.

When deployed to MarkLogic ./plugins is equivalent to the root URI (/), so a library module at ./plugins/my-directory/my-lib.xqy would be loaded into the modules database as /my-directory/my-lib.xqy.

plugins/entities

  entities
  ├─ entity1
  │  ├─ input
  │  │  ├─ inputflow1
  │  │  │  ├─ content.(sjs|xqy)
  │  │  │  ├─ headers.(sjs|xqy)
  │  │  │  ├─ main.(sjs|xqy)
  │  │  │  └─ triples.(sjs|xqy)
  │  │  ├─ ...
  │  │  └─ inputflowN
  │  └─ harmonize
  │     ├─ harmonizeflow1
  │     │  ├─ collector.(sjs|xqy)
  │     │  ├─ content.(sjs|xqy)
  │     │  ├─ headers.(sjs|xqy)
  │     │  ├─ main.(sjs|xqy)
  │     │  ├─ triples.(sjs|xqy)
  │     │  └─ writer.(sjs|xqy)
  │     ├─ ...
  │     └─ harmonizeflowN
  ├─ ...
  └─ entityN

plugins/entities: This directory contains your entity definitions. An entity is a domain object like Employee or SalesOrder. Each entity directory contains two subdirectories: input and harmonize. DHF has custom logic to handle the deployment of this directory to MarkLogic.
NOTE: The entities directory is reserved for data hub use and is treated as a special case by the deploy process.
plugins/entities/entity/input: The input subdirectory contains all of the input flows for a given entity. Input flows are responsible for creating an XML or JSON envelope during content ingest. This directory contains one server-side module for each part of the envelope: content, headers, and triples. You may also optionally include a REST directory that contains custom MarkLogic REST extensions related to this input flow.
plugins/entities/entity/input/content.(sjs|xqy): The server-side module (XQuery or JavaScript) responsible for creating the content section of your envelope.
plugins/entities/entity/input/headers.(sjs|xqy): The server-side module (XQuery or JavaScript) responsible for creating the headers section of your envelope.
plugins/entities/entity/input/main.(sjs|xqy): The server-side module (XQuery or JavaScript) responsible for orchestrating your plugins.
plugins/entities/entity/input/triples.(sjs|xqy): The server-side module (XQuery or JavaScript) responsible for creating the triples section of your envelope.
plugins/entities/entity/harmonize: The harmonize subdirectory contains all of the harmonize flows for a given entity. Harmonize flows are responsible for creating an XML or JSON envelope during content harmonization. This directory contains one server-side module for each part of the envelope: content, headers, and triples. It also contains collector and writer modules as described below. You may also optionally include a REST directory that contains custom MarkLogic REST extensions that are related to this input flow.
plugins/entities/entity/harmonize/collector.(sjs|xqy): The server-side module (XQuery or JavaScript) responsible for returning a list of things to harmonize. Harmonization is a batch process that operates on one or more items. The returned items should be an array of strings. Each string can have any meaning you like: uri, identifier, sequence number, etc.
plugins/entities/entity/harmonize/content.(sjs|xqy): The server-side module (XQuery or JavaScript) responsible for creating the content section of your envelope.
plugins/entities/entity/harmonize/headers.(sjs|xqy): The server-side module (XQuery or JavaScript) responsible for creating the headers section of your envelope.
plugins/entities/entity/input/main.(sjs|xqy): The server-side module (XQuery or JavaScript) responsible for orchestrating your plugins.
plugins/entities/entity/harmonize/triples.(sjs|xqy): The server-side module (XQuery or JavaScript) responsible for creating the triples section of your envelope.
plugins/entities/entity/harmonize/writer.(sjs|xqy): The server-side module (XQuery or JavaScript) responsible for persisting your envelope into MarkLogic.
plugins/entities/entity/harmonize/REST: Items that used to be here have been moved to src/main/ml-modules in DHF 4.x.

plugins/mappings

  mappings
  ├─ mapping1
  │  ├─ mapping1-0.mapping.json
  │  ├─ ...
  │  └─ mapping1-N.mapping.json
  ├─ ...
  └─ mappingN

This directory contains model-to-model mapping configuration artifacts that can be used to configure an input flow. For details, see Using Model-to-Model Mapping.

plugins/mappings/mapping: This directory contains all versions of a given model-to-model mapping. The name of the directory is the same as mapping name. For details, see For details, see Using Model-to-Model Mapping.
plugins/mappings/mapping/mapping-version.json: This JSON file is a model-to-model mapping configuration file. QuickStart creates a new version each time you modify a mapping. See Using Model-to-Model Mapping.

src/main

  src
  └─ main
     ├─ entity-config
     │  └─ databases
     ├─ hub-internal-config
     │  ├─ databases
     │  ├─ schemas
     │  ├─ security
     │  ├─ servers
     │  └─ triggers
     ├─ ml-config
     │  ├─ databases
     │  ├─ entities.layout.json
     │  ├─ security
     │  ├─ servers
     │  └─ triggers
     ├─ ml-modules
     └─ ml-schemas

NOTE: As of DHF 4.1.0, the internal structures of all configuration directories align with that of ml-gradle and should work as documented in the ml-gradle documentation.

src/main/entity-config

  entity-config
  ├─ final-entity-options.xml
  ├─ staging-entity-options.xml
  └─ databases
     ├─ final-database.json
     └─ staging-database.json

This directory contains two options files and two database configuration files for staging and for final. These files can be modified to configure indexes.

src/main/hub-internal-config

  hub-internal-config
  ├─ databases
  │  ├─ job-database.json
  │  ├─ staging-database.json
  │  ├─ staging-schemas-database.json
  │  └─ staging-triggers-database.json
  ├─ schemas
  ├─ security
  │  ├─ privileges
  │  │  ├─ dhf-internal-data-hub.json
  │  │  ├─ dhf-internal-entities.json
  │  │  ├─ dhf-internal-mappings.json
  │  │  └─ dhf-internal-trace-ui.json
  │  ├─ roles
  │  │  ├─ data-hub-role.json
  │  │  └─ hub-admin-role.json
  │  └─ users
  │     ├─ data-hub-user.json
  │     └─ hub-admin-user.json
  ├─ servers
  │  ├─ job-server.json
  │  └─ staging-server.json
  └─ triggers

This directory contains subdirectories and JSON files that represent the minimum configuration necessary for DHF to function. Do not edit anything in this directory. If you need to override a configuration in this directory, create a file with the same name and directory structure under the ml-config directory and add any properties you’d like to override.

Each of the above JSON files conforms to the MarkLogic REST API for creating the following:

src/main/ml-config

  ml-config
  ├─ databases
  │  ├─ final-database.json
  │  ├─ final-schemas-database.json
  │  ├─ final-triggers-database.json
  │  └─ modules-database.json
  ├─ entities.layout.json
  ├─ security
  │  ├─ privileges
  │  ├─ roles
  │  └─ users
  ├─ servers
  │  └─ final-server.json
  └─ triggers

This directory contains additional subdirectories and JSON files used to configure your DHF project. You can add custom modules and transforms, as well as other configuration assets, in this directory.

The following files are found in the ml-config directory only:

final-database.json
final-schemas-database.json
final-triggers-database.json
modules-database.json
final-server.json

IMPORTANT: Starting with DHF 4.1.0, custom triggers must be added to ml-config/databases/(database name)/triggers. (In previous versions, triggers were added to hub-internal-config/triggers or ml-config/triggers.) See ml-grade Project Layout for more information on triggers.

src/main/ml-modules

This directory is the default ml-gradle location for artifacts to be deployed to the modules database.

src/main/ml-schemas

This directory contains your project’s schemas which are loaded by ml-gradle.

.tmp

This directory contains temporary hub artifacts.

Project Directory Structure in Previous DHF Releases

DHF 4.0.x