Back to Product Wiki

 

Overview

This article will go over the two methods of uploading data for a File Import profile. 

For an article on creating and configuring a File Import profile, please navigate to the Creating and Configuring a File Import profile article.

Download from FTP site

If you have an FTP server at your disposal that receives data from your sensors then iTwin IoT can fetch data from that server at a fixed interval, with the interval being fixed at every 10 minutes. 

There are a number of options available to the user to configure this, they can select how the files are handled after import, how to handle filename matching and more.

 Upload to iTwin IoT using FTP client

The automated import feature allows the user to configure an FTP (file transfer protocol) to enable file transfer from a 3rd party sensor or the users computer (FTP Client) over to iTwin IoT's FTP server. This will enable you to have your File Import run automatically every time your sensor sends a new file to the FTP folder that was created once you created a File Import profile.

 

In order for the FTP-powered automated import tool to function, please ensure that Ports 20, 21, and 22 are open to transfer.iiot-services.bentley.com on the local network.

Upload from computer

This method is for uploading data manually, this is useful for example manual measurements in the field, uploading historical data, fixing data gaps and etc.

Automatic File transports

The File Import supports two types of Automatic File transports as it stands, both are based on the FTP transfer protocol. One allows iTwin IoT to fetch data from a server and the other allows iTwin IoT to receive data via FTP.

Download from FTP site

FTP Connection

When configuring your file import to fetch data from an FTP server of your choosing the first step is to configure the FTP connection itself. You can do that by clicking the Add FTP Connection option.

This will open up a window that allows you to configure and test your FTP connection. 

The settings are as follows:

FTP Host: This is the URL or IP address of your FTP server. For example: ftp.company.com or just data.company.com 

Port: This is the FTP port, by default it is 21 but you can change this if your server uses an alternative port. 

Mode: This enables you to choose FTPS or SFTP.

Remote Path: This is the remote path for accessing the files. If your user points at the root then you can set a default sub-path here, for example /UK/IPI/ or /NL/IPI/ . If you are unsure then leave this empty. 

Username and Password: This is the username and password for your FTP user. Take care that this user has to have view, edit and create privileges on your FTP server so that the iTwin IoT FTP download function works properly.

Remote server file management:

Under this settings there are three options. 

Do nothing: This means that once iTwin IoT fetches the datafile and imports the data from it it leaves it alone, this is useful if you just have one datafile that your datalogger appends data to.

Delete after downloading: This means that once iTwin IoT fetches the datafile and imports the data from it it deletes the file. This can be useful if you have a fast datastream with many files and you want to clean up your server as you import data into iTwin IoT.

Archive to subdirectory after downloading: This means that once iTwin IoT fetches the datafile and imports the data from it it moves that file into an archive sub-directory within the directory where that datafile is located. This is mostly useful when using incremental file import, that is one file contains one or a couple of datalines and the datalogger creates a new file for each measurement.

 

Filename matching

Sometimes your folder with your datafile has multiple files from other dataloggers that you do not want to import.

Filename matching allows you to define a matching criteria to your Filename and to ignore files that don't match that pattern.

Here the folder contains 4 files from 4 different loggers but the import should only pick up SingleFile1.csv. In this case adding a SingleFile1 matching text will only match it with any file with that name in it. 

As you are typing in the matching text the display of files will change according to your filtering name. Once you are done you click Done and now that Filename pattern will be stored for this File Import profile. 

 

Upload to iTwin IoT using FTP client

FTP Setup

There are two workflows for importing data via FTP, the process on iTwin IoT (FTP server) is the same but the configuration of the FTP client is different. 

Workflow 1 - upload from users PC

In order to utilize the FTP, an FTP client that supports FTPS must be installed on the user's computer. iTwin IoT recommends the use of Filezilla, but many other FTP clients may be used. This help article will illustrate the workflow within the Filezilla FTP client. If you would like to use another FTP client, please reach out to the iTwin IoT support team for confirmation of whether your FTP client of choice is compatible with the FTPS requirements of the iTwin IoT platform. 

This workflow isn't necessarily only for automating an import, you can use this for manual measurements that you can upload via Filezilla if you do not want to log into iTwin IoT every time you want to upload your readings manually. 

Start by downloading the Filezilla FTP client (download link: https://filezilla-project.org/). After downloading and completing the initial setup of Filezilla, open up the application. You will navigate back to the application shortly.

Workflow 2 - upload from 3rd party sensor/logger/gateway

This workflow is useful for when you have a 3rd party sensor that is not directly supported in iTwin IoT. What this enables you to do is to have that sensor or gateway send any new data automatically to the FTP folder that is created for your File Import. 

The process of creating the File Import itself is the exact same, but instead of using Filezilla to upload the files you configure the sensor/logger/gateway to send the data via FTP or FTPS to your relevant FTP folder in iTwin IoT.

Accessing FTP Information

Once you have created a file import as shown in the Creating and Configuring a File Import article you will be able to upload data automatically to the FTP folder that was created for that File Import. In the image below you can see the FTP host/server URL, the username, password and directory for this particular File Import Profile. This is the FTP information used for sending data via FTP to your File Import profile. 

Filezilla FTP setup (Workflow 1)

1. Host, Username, Password Inputs- The FTP credential inputs will be at the top of the page. Input the data collected from iTwin IoT above into the corresponding inputs labelled alphabetically. A, b, and c will only be used in this initial configuration step. You can match the host (a), username (b), and password (c) with the information collected above. A port number does not need to be input.

 

 

An example of this information entered into the QuickConnect bar of Filezilla is displayed below.

 

 

Once the FTP is configured, click the "Quickconnect" button. A status message should appear at the top saying the connection is successful, and you will see a list of directories appear on the right hand, Remote site", side of the Filezilla page. Proceed to the next section if the connection succeeds.

 If the connection fails, check the input information from above and ensure there are no firewall settings on your network or computer that prohibit FTP usage. This information can be found at the top of the page or in this support article. If you've checked all of these possible issues, please contact our support team for assistance.

 FTP Connection and File Transfer

After clicking the "Quickconnect" button, Filezilla will then attempt to connect between the iTwin IoT remote host and the user's computer. 

 

 

1. Status Bar- The Status Bar provides a log of operations for the FTP. If the connection procedure above was followed correctly, the Status notes bar will end with the note "Status: Directory listing of "/" successful". Sections 2, 3, 4, and 5 will populate once the connection is successful.

2. Local Host Folders- The user's computer drives and folders will be displayed on the upper-left window. Navigating through drives and folders within this window will change the sub-folders and files displayed in the lower-left window.

3. Remote Host Folders- The iTwin IoT file directory for the FTP documents module will be displayed in the upper-right corner. The selected file should be the file with the Username displayed as the folder name, typically an alphanumeric string. Clicking on this username will display the sub-folders within, also called the directories, which are the randomly generated folder names from the Import profile creation step earlier. Note: The username and password will be shared for all Import Profiles created within a given project. All Import Profiles created within an iTwin IoT project will be visible as a folder under the username in this window.

4. Local Host Sub-Folders and Files- The user's computer sub-folders and files, as selected in window 2, will be displayed on the lower-left window. Folders and files from this window can be dragged and dropped to the right into the Remote Host folders, but a Directory must be created first in Step 5. When dragging and dropping, confirm that a green "+" box appears on the Remote Host Directory folder you are transferring the file into to ensure success.

5. Remote Host Sub-Folders and Files- Sub-folders, or directories, are automatically generated when creating an Import Profile in iTwin IoT. To ensure that data is going into the correct directory for the File Import profile, please reference the Automation tab in iTwin IoT. The image below highlights the location of the auto-generated directory name. Note: All Import Profiles created within an iTwin IoT project will be visible as a folder under the username in this window.

 

 

 

Files can be uploaded into the directory to make them visible in the Import Profile by drag and drop or by highlighting the directory and file/files then clicking upload. When dragging and dropping, confirm that a green "+" box appears on the Local Host folder you are transferring the file into to ensure success. The screenshot below is a highlighted example screenshot illustrating where the directory is located in Filezilla.