File
This plugin pulls metadata from a previously generated file. The file sink can produce such files, and a number of samples are included in the examples/mce_files directory.
CLI based Ingestion
Install the Plugin
The file
source works out of the box with acryl-datahub
.
Starter Recipe
Check out the following recipe to get started with ingestion! See below for full configuration options.
For general pointers on writing and running a recipe, see our main recipe guide.
source:
type: file
config:
# Coordinates
filename: ./path/to/mce/file.json
sink:
# sink configs
Config Details
- Options
- Schema
Note that a .
is used to denote nested fields in the YAML recipe.
Field | Description |
---|---|
path ✅ string | File path to folder or file to ingest, or URL to a remote file. If pointed to a folder, all files with extension {file_extension} (default json) within that folder will be processed. |
aspect string | Set to an aspect to only read this aspect for ingestion. |
count_all_before_starting boolean | When enabled, counts total number of records in the file before starting. Used for accurate estimation of completion time. Turn it off if startup time is too high. Default: True |
file_extension string | When providing a folder to use to read files, set this field to control file extensions that you want the source to process. * is a special value that means process every file regardless of extension Default: .json |
read_mode Enum | Default: AUTO |
The JSONSchema for this configuration is inlined below.
{
"title": "FileSourceConfig",
"type": "object",
"properties": {
"path": {
"title": "Path",
"description": "File path to folder or file to ingest, or URL to a remote file. If pointed to a folder, all files with extension {file_extension} (default json) within that folder will be processed.",
"type": "string"
},
"file_extension": {
"title": "File Extension",
"description": "When providing a folder to use to read files, set this field to control file extensions that you want the source to process. * is a special value that means process every file regardless of extension",
"default": ".json",
"type": "string"
},
"read_mode": {
"default": "AUTO",
"allOf": [
{
"$ref": "#/definitions/FileReadMode"
}
]
},
"aspect": {
"title": "Aspect",
"description": "Set to an aspect to only read this aspect for ingestion.",
"type": "string"
},
"count_all_before_starting": {
"title": "Count All Before Starting",
"description": "When enabled, counts total number of records in the file before starting. Used for accurate estimation of completion time. Turn it off if startup time is too high.",
"default": true,
"type": "boolean"
}
},
"required": [
"path"
],
"additionalProperties": false,
"definitions": {
"FileReadMode": {
"title": "FileReadMode",
"description": "An enumeration.",
"enum": [
"STREAM",
"BATCH",
"AUTO"
]
}
}
}
Code Coordinates
- Class Name:
datahub.ingestion.source.file.GenericFileSource
- Browse on GitHub
Questions
If you've got any questions on configuring ingestion for File, feel free to ping us on our Slack.