Batch Files
Last updated
Last updated
1. Go to the Resources page and click the Add Resource button.
2. In the "Select a data source" step, navigate to Flat File > Batch Files.
3. Click Select Files to upload your files. Here you can upload a bunch of files at a time. There is no limit for the number of rows to upload. Before uploading files, please note that:
All data files should have the same schema (the same number of columns and the same data type for each column). However, the column names can be different.
Currently, Batch Files only supports .csv, .tsv, and .txt files. If you want to upload Excel (.xlsx) files, please choose Single File instead in Step 2.
The maximum number of files you can upload at a time is up to 100 files.
4. Then, in CSV Preview Setup, you can preview the first file. In this step, you can change the resource name or check if your file is parsed into the correct format. If your file doesn't have column names, you can switch to Safe mode to add a header row.
If all files don't have a header row and you don't add a header row in CSV Preview Setup, all files lose the first row. This is because the system will consider the first row as a header row automatically.
5. Click Upload and you will see the files are uploading in the bottom-right window. It may take a few minutes to several hours to upload all data files depending on the data size and the number of your files.
6. Once your Batch Files resource is built up, you can click the resource to enter into File Management. It shows the list of files that you just uploaded and their detailed information for each file. Here you can also add new files or delete some of the files.
Once you build up a Batch Files resource, you can go to File Management to add or delete files by clicking the dropdown menu next to your Batch Files resource.
If you have new files and want to merge them into your current Batch Files, you can just click Add Files and choose data files to upload. Remember the new files should have the same schema as the one in the other files.
In File Management, you can also delete some of the files in your Batch Files. Just navigate to the file that you want to delete and click the Delete button.
If you add or delete files in your Batch Files resource, you can go to your project to retrieve the latest records without rebuilding your data pipeline.
When you create a Batch Files Resource and import data to a project, you will find a new column called _acho_meta_name that doesn't exist in your dataset. This column is generated automatically when you upload files as a Batch Files. This can help you identify which rows originated from which data files and when these files are imported.
If you want to extract information from this column, you can use the Split action to separate this column or write a regular expression to extract a specific string in Formula.
This problem may happen when some of your data files don't have a header. If you face this situation, please follow the steps below to reupload your data:
Go to Resource and choose Batch Files to upload your files again.
In Data Preview Setup, turn on the Safe Mode.
Click Add a Header Row.
Now, the system will add a header for all of your data files automatically.
There are two options to tackle this situation:
Option 1: Open the files consisting of headers, and remove their headers. Then, follow the steps in Situation 1 to re-upload these files.
Option 2: If you are not sure which files have headers and which ones don't, you can follow the steps in Situation 1 to re-upload files and add a header row (no matter whether a header already exists in Data Preview Setup). Then, import the Batch Files to a project and use Filter to filter out the header rows.
