Overview

The automated import feature allows the user to configure an FTP (file transfer protocol) to enable file transfer between a remote host (the iTwin IoT cloud platform) and the local host (a user's computer) for sensor data not integrated into ITwin IoT via telemetry connecitons. This article will guide the user through the setup and operation of the FTPS between the iTwin IoT app and the user's computer.

NOTE- In order for the FTP-powered automated import tool to function, please ensure that Ports 20 and 21 are open to transfer.iiot-services.bentley.com on the local network.

FTP Setup

In order to utilize the FTP, an FTP client that supports FTPS must be installed on the user's computer. iTwin IoT recommends the use of Filezilla, but many other FTP clients may be used. This help article will illustrate the workflow within the Filezilla FTP client. If you would like to use another FTP client, please reach out to the iTwin IoT support team for confirmation of whether your FTP client of choice is compatible with the FTPS requirements of the iTwin IoT platform.

 

Start by downloading the Filezilla FTP client (download link: https://filezilla-project.org/). After downloading and completing the initial setup of Filezilla, open up the application. You will navigate back to the application shortly.

 

Beginning Automated Import Setup and Accessing FTP Information

1. Navagating to the Data Module- Click on the Data Icon along the left module navigation sidebar to navigate to the Data Module. 

2. Navigating to File Import Module- Click on File Import on the ribbon that appears on the left of the page after selecting Data. 

3. New Import Profile- Click the blue "+Import" icon opens the File Import configuration page, allowing the user to configure a new import profile (seen below).

 

 

3. FTP Information- The information required to configure the FTP will be available in the following setup page, located under the "Automation" tab. These credentials can be copied from this page into the FTP software during setup as will be explained in the following section. Take note of the Directory name in the box. This will be necessary for pushing files to the correct Import Profile via the FTP in later steps. NOTE: Do not close this window or save at this step. After configuring the FTP connection, you must navigate back to this page to finish setup in iTwin IoT.

 

 

FTP Setup

Navigate back to the Filezilla page.

1. The FTP credential inputs will be at the top of the page. Input the data collected from iTwin IoT above into the corresponding inputs labelled alphabetically. A port number does not need to be input.

 

 

An example of this information entered into the QuickConnect bar of Filezilla is displayed below.

 

 

Once the FTP is configured, click the "Quickconnect" button.

 

FTP Connection and File Transfer

After clicking the "Quickconnect" button, Filezilla will then attempt to connect between the iTwin IoT remote host and the user's computer. 

 

 

1. Status Bar- The Status Bar provides a log of operations for the FTP. If the connection procedure above was followed correctly, the Status notes bar will end with the note "Status: Directory listing of "/" successful". Sections 2, 3, 4, and 5 will populate once the connection is successful.

If the connection is not established, confirm that the connection information was input correctly and that your local network firewall settings allow FTP.

2. Local Host Folders- The user's computer drives and folders will be displayed on the upper-left window. Navigating through drives and folders within this window will change the sub-folders and files displayed in the lower-left window.

3. Remote Host Folders- The iTwin IoT file directory for the FTP documents module will be displayed in the upper-right corner. The selected file should be the file with the Username displayed as the folder name, typically an alphanumeric string. Clicking on this username will display the sub-folders within, also called the directories, which are the randomly generated folder names from the Import profile creation step earlier. Note: The username and password will be shared for all Import Profiles created within a given project. All Import Profiles created within an iTwin IoT project will be visible as a folder under the username in this window.

4. Local Host Sub-Folders and Files- The user's computer sub-folders and files, as selected in window 2, will be displayed on the lower-left window. Folders and files from this window can be dragged and dropped to the right into the Remote Host folders, but a Directory must be created first in Step 5. When dragging and dropping, confirm that a green "+" box appears on the Remote Host Directory folder you are transferring the file into to ensure success.

5. Remote Host Sub-Folders and Files- Sub-folders, or directories, are automatically generated when creating an Import Profile in iTwin IoT. To ensure that data is going into the correct directory for the File Import profile, please reference the Automation tab in iTwin IoT. The image below highlights the location of the auto-generated directory name. Note: All Import Profiles created within an iTwin IoT project will be visible as a folder under the username in this window.

 

 

 

Files can be uploaded into the directory to make them visible in the Import Profile by drag and drop or by highlighting the directory and file/files then clicking upload. When dragging and dropping, confirm that a green "+" box appears on the Local Host folder you are transferring the file into to ensure success. The screenshot below is a highlighted example screenshot illustrating where the directory is located in Filezilla.

 

 

Finishing Automated File Import Profile Configuration

 

Navigate back to the iTwin IoT application window in the browser. If the FTP setup has been successful, data should now be previewed within iTwin IoT as seen in the right side of the example image below. Additional configuration of the Import profile can then begin.

 

 

 

1. Import Name- Name the Import profile.

2. Parsing- Parsing rules may need to be adjusted for the file data to be correctly imported. Parsing rules are automatically applied to the data preview on the right, allowing the user to ensure that the correct parsing rules are selected for the file to be imported. When configured correctly, the data on the right-side preview should look similar to the image below. If data is not being parsed correctly, please reference the Advance Parsing section information below.

 

 

2a. Header Label Row- Select the row of the spreadsheet with header information describing the spreadsheet columns. This selection will aid the user in configuring the rest of the import profile by populating the Header Labels in the remaining portions of the Import Profile. The selected Header Label Row will be highlighted in green on the preview window to the right. If no header row is present in the spreadsheet, select "None".

2b. Timestamp Selection- The timestamp selection options will be populated by the Header Label Row cells. Select the checkbox next to the correct column holding Timestamp data. The selected column will then be highlighted light gray in the preview window. Ensure that the timestamp format auto-populated after selecting the correct column matches the data preview.

 

An example of the timestamp format properly matched to the spreadsheet data is available below.

 

 

If the formatting is incorrect, the timestamp column will be highlighted red, as shown in the image below.

 

 

2c. Time Zone- Select the time zone via dropdown of the time stamps for the data being imported.

2d. Advanced Parsing- If the default parsing settings do not parse the data correctly, the Advanced Parsing settings may need to be utilized to properly parse the data for import.

 

 

Delimiter- Select the delimiter the file is using as the field qualifier.

Text Qualifier- Select the qualifier the file is using for text.

Text Encoding- Define the text encoding manner of the file.

Header Units Row- If a header unit row is present, select the row from the dropdown.

 

3. Sensors- The sensor configuration allows the user to select the Sensor ID column within the import file, to select the sensor within iTwin IoT to import the data into or create a new sensor to store the import data, and define the metric and units of the data being imported. The sensor configuration step is also where you will configure the import of multiple sensors, if required. This example will cover the configuration of a single sensor.

 

 

3a. Sensor ID- Select the data column housing the sensor name or names. Once selected, the column will be highlighted light grey on the data preview. 

 

 

3b. Sensor Selection or Sensor Creation- The sensors selection allows the user to select where the data will be imported. If there are multiple sensor names in the selected sensor column, multiple sensor configuration windows will populate, as illustrated below.

The two initial options for each sensor available are "Existing" for sensors already added to iTwin IoT or "New" to create a new sensor within iTwin IoT for the import of the data.

 

Existing Sensor Configuration- An existing sensor is a sensor that has been added to the platform when establishing a new IOT connection, adding an IOT device/sensor, or a previously created import sensor. Clicking the "Existing"  button opens a sensor selection window, illustrated below, allowing the user to select from existing sensors. Clicking a sensor in the left column will select it, moving it to the selected sensor column, and marking it as the sensor the data will be imported to.

 

 

After navigating back to the Import profile, the user will need to match import data to the required data for the selected existing sensor, as illustrated below. Note: If the import data doesn't match the data requirements on the left, the import data is likely a calculated metric. Raw sensor metrics must be imported for existing sensors.

 

 

New Sensor Configuration- The "New" option allows the user to create a new sensor within iTwin IoT to provide an import option for data or sensors not associated with an IoT-connected sensor. 

 

 

 

1. Sensor Name- By default, the sensor name will be the sensor name provided in the Sensor row configured earlier. This can be changed at this time and will be the sensor name used for the sensor in iTwin IoT.

2. Device Assignment- The device is the middle of the hierarchy of the IoT network. The device groups sensors by connection, or allows the user to group import sensors manually, within the Connectivity Module. There are two options available for device assignment when importing data.

The "Existing" selection option opens a dropdown, illustrated below, where the user can select a device for the imported sensor data to be imported into.

 

 

The "New" option allows the user to create and name a new Device for the import.

 

3. Sensor Type Selection- The sensor type selection dictates whether the data will be imported into an existing sensor type (defined metrics based on sensors) or into a newly created custom sensor (allowing the user to select their own data metrics for import).

 

Existing Type Sensor- Selecting Existing type reveals a dropdown allowing the user to select a sensor type.

 

 

After selecting an existing sensor type and navigating back to the Import profile, the user will need to match import data to the required data for the selected existing sensor, as illustrated below. Note: If the import data doesn't match the data requirements on the left, the import data is likely a calculated metric. Raw sensor metrics must be imported for existing sensors.

 

 

Custom Sensor- The custom sensor selection allows the user to create any metric they want by inputting a metric and unit. Additional metrics can be added by clicking the "+Metric label" button.

 

 

3. Metric and Unit Selection- After configuring the custom sensor, the custom metrics must be matched to the data import file columns.

 

 

4. Alarms

Alarms can be configured to inform the user if a file encountered upload issues or if the automated import stopped working.

 

 

Success Rate- Setting the success rate requirement will enable an alert if the stipulated parsing minimum is not met. 

 

 

Maximum Timeout- The maximum timeout is defined as the amount of time allowed between automated file uploads before an alarm is triggered. This allows the user to know if there has been an interruption in the FTP automation. Option range from 1 hour to 90 days.

 

 

 

Starting/Saving Import and Import Diagnostics

Once the configuration is complete, the "Save & Import" button becomes available. This will enable the automated import. Alternatively, the Import Profile can just be saved for later use.

 

 

Clicking the "Save and Import" button will enable the import and direct the user to an import summary page, as illustrated below.

 

 

Checking Status of Import

 

The status of the automated import function can easily be checked From the File Import main page, the latest attempted file import can be tracked in the "File Received" column for the automated import.

 

 

More in-depth information about the import can be accessed by clicking on the name of the Import Profile from the main File Import page, as shown below. The pencil icon on this page can also be used to edit the Import Profile configuration.