Creating and Configuring a File Import profile


 

Back to Product Wiki

 

Overview

The File Import Module allows users to manually import data or set up an automated data import stream. This article will cover the process of creating a Manual Import profile and importing data into iTwin IoT. The manual import consists of bringing one file in at a time via the created import profile. If configured correctly, data for multiple sensors of the same type can be imported from a single file. The process of creating a manual import or automatic import is the exact same, you will always have the option of sending data automatically to a File Import.

This article will cover:

Navigating to File Import Tool

Clicking the "Sensors" option on the left toolbar will present you with these options. You click the "File Import" option to access this feature.

 

 

You will now be in the File Import Main page, as shown below. Once on this page you can create a new import by clicking the "+ Import" button in the upper left. 

 

 

 

Selecting the import profile

When you create a new File Import you currently have three options for what type of data is being imported.
The types are as follows:

Time Series Data - This is for a wide variety of Time series data for many different file types and data formats. Currently the File Import only supports .csv, .txt and .dat file types.

Measurand Shapearray (SAA) - This is a specific import type for Measurand's SAA sensors. This can be a file that is coming from a Campbell Scientific logger or an RST logger for example. You can upload the file first and then have the loggers send all future data to the FTP location that this file import will create. 

DGSI Manual Inclinometer - This is a specific import for DGSI's Manual Inclinometer data. This data can either be uploaded manually directly through iTwin IoT's web interface or via Filezilla from your computer using the FTP location that this file import will create. 

Time Series Data Import configuration

1. New Import Profile- Once on the File Import main page, clicking the blue "+Import" icon opens the File Import configuration page, followed by the Import type selection, allowing the user to begin configuring a new import profile (example seen below).

 

 

Please take note of the difference between a Row based sensor file and Column based sensor file. 

A Row based sensor file is used for this example but it is not always the case. A row based sensor file means that there is a column that contains the info on what sensors data is in that row. Then all other rows will represent the timestamp for the observation of that sensor and the metrics for that sensor. In this case the "Sensor names column" is a required selection. 

A Column based sensor file means that each column contains one metric, you can then decide if your sensor contains only one metric or multiple. You can configure this however you like. This means that there is no column in the file that defines what the sensor is. The columns name in the header defines that. For this case the "Sensor names column" option turns into "Sensor names row" and becomes optional.

   

 

2. Import Name - Name the Import profile.  

3. File Upload - A file can be dragged and dropped into the upload area or you can click the blue "Choose a file" button to open a file-selection window. Select the file to be imported. Once a file has been selected, the window will be populated with the file data on the right side of the screen, Illustrated below, and the additional configuration of the import can be completed.

 

 

4. Time Stamp Column - Select the column that contains the timestamp for the sensor data (date and time of data collection). Once a selection is made, a box will appear below the selected column, and you will need to configure the timestamp. For more information about Timestamp configuration, please visit the Timestamp Configuration knowledge article.

 

 

5. Time Zone- Select the time zone of the data in the import file. (It is important to remember that data in iTwin IoT will be stored in a certain time zone, but data visualization in iTwin IoT will convert that data to reflect the time zone selected on the computer of whomever is viewing the data.)

6. Advanced Parsing- Parsing rules may need to be adjusted to ensure the data is imported correctly. When configured correctly, the data on the right side preview should look similar to the image below. Parsing rules are automatically applied to the data preview on the right, allowing the user to ensure that the correct parsing rules are selected for the file to be imported. If the data in the data preview window doesn't look correctly parsed, you may need to adjust the Advance Parsing settings, explained below.

 

 

6a. Delimiter - Select the delimiter the file is using as the field qualifier from the dropdown options. Comma is the default selection for iTwin IoT. 

6b. Text Qualifier - Select the qualifier the file is using for text.

6c. Text Encoding - Define the text encoding manner of the file.

 

Sensor Configuration 

The sensor configuration(7) section allows you to map the data from your import file to existing sensors within your project or to create new sensors to hold the imported data. This is also where you will define the sensor type and match the spreadsheet data to the corresponding metrics and metric units in iTwin IoT. 

 

 

7a. File Direction - It specifies the orientation of the sensor data within the file. You can choose between two options:

Rows - Each sensor's data is spread across one or more rows. 

Columns- Each sensor's data is organized in one or more columns.

7b. Sensor Names Column - This mandatory dropdown menu allows you to select the column in your file that contains the names of the sensors. This is most common if there are multiple sensors on a single file. Once a selection is made, the "Sensors" section below will be populated with a configuration box for each sensor identified in the selected column. In the screenshot, "Column 1" is selected. This is not a required option if you are using a Column Based file direction.

7c. Metrics and Units Rows - These optional dropdown menus allow you to specify if there are particular rows in your file that define the metrics (e.g., Temperature, Pressure) and their corresponding units (e.g., °C, kPa).

Metrics row - Select the row containing the names of the metrics.

Units row - Select the row containing the units for the measurements.

7d. Sensors - After configuring the "Sensor names column," this section will list the individual sensors found in your file. For each sensor identified, you have two configuration options: "Existing" or "New". "Existing" option is chosen to map the imported data to a sensor that already exists in the system. "New" option is chosen to create a new sensor record in the system with this data.

Additionally, a "Batch-create" button is available, which allows for the creation of multiple new sensors at once, streamlining the process when dealing with files containing data from many new sensors.

 

  • Existing Sensor - An existing sensor is a sensor that has been added to the platform when establishing a new IOT connection, adding an IOT device/sensor, or a previously created import sensor. Clicking the "Existing"  button opens a sensor selection window, shown below, allowing the user to select from existing sensors. Clicking a sensor in the left column will select it, moving it to the selected sensor column, and marking it as the sensor the data will be imported to. After navigating back to the Import profile, the user will need to match import data to the required data for the selected existing sensor, as illustrated below. 

 

  • New Sensor - The "New" option allows the user to create a new sensor within iTwin IoT to provide an import option for data or sensors not associated with an IoT-connected sensor. Selecting "New" will open a popout window similar to the one below. 

 

 

    • 1. Sensor Name -  In this field, you must enter a unique name for the sensor you are creating. 
    • 2. Start and End Columns - These fields specify the range of columns in your data that the sensor will apply to. In this example, with both set to "4", the sensor will only monitor the 4th column. 
    • 3. Sensor type - This section determines the kind of sensor you are creating. "Predefined" option allows you to choose from a list of pre-existing sensor templates, which is useful for common monitoring tasks. "Custom" option allows you to build a new sensor from scratch, defining your own rules and parameters.

a. Predefined

        • Select predefined sensor type - The dropdown menu where you would choose one of the available predefined sensor types, as shown in above image.

                       b. Custom

        • Map metrics and units - This section is for defining the metrics for columns, which were specified in the "Start and end columns" fields above. If you select a range of columns (e.g., 4 to 6), you would see separate mapping fields for each column in that range, as shown in following image. 

I. Metric - This input field is where you name the measurement being captured in the column. For example, if column 4 contains temperature readings, you would enter "Temperature" here.

II. Unit - This input field is where you specify the unit of measurement for the metric you just defined. For the "Temperature" example, you might enter "Celsius," "Fahrenheit," or "Kelvin."

 

 

Batch-create sensors

It allows you to create multiple sensors simultaneously using a template and patterns in your data. It automates the process by applying a common configuration to data from specified columns, which saves significant time and effort. 

    • 1. Sensor naming pattern - This section helps determine how each sensor in the batch will be named.
    • 2. Always use names from sensor names column - When this toggle is switched on (as it is in the image), the system will automatically name each new sensor using the values from a designated "sensor names column." The blue information box at the bottom clarifies that for this operation, it will use the IDs from Column 1 as the names.
    • 3. Sensor name base - This field is currently disabled because the toggle is on. If you were to switch the toggle off, the "Sensor name base" field becomes active as shown below. You can then enter a base name (e.g., "PressureSensor"), and the system will automatically create numbered sensors like "PressureSensor-1," "PressureSensor-2," and so on, for each row it processes. 
    • 4. Start and end columns - This functions just like in the single sensor creation, defining which column(s) will be monitored. Here, it is set to create sensors for the data in Column 4.
    • 5. Sensor type - You can choose either a Predefined or Custom sensor definition. 

a. Predefined - Creates sensors based on an existing template. 

I. Refresh list - Updates the list of available predefined sensor types.

II. Select predefined sensor type - A mandatory dropdown where you select the template that will be used for all the sensors in this batch. 

b. Custom - It allows you to create a new sensor definition from scratch, which will be applied to all sensors in the batch. Selecting this option reveals the "Map metrics and units" section below.

        • Map metrics and units - This section appears when you choose the "Custom" sensor type. It allows you to define what the data represents for all sensors being created in the batch.

I. Refresh list - Updates the mapping interface.

II. Always use metric from metrics row - This toggle is currently off. When off, the "Metric" and "Unit" you define manually in the "Column 4" section below will be applied uniformly to every sensor created in this batch. If this were toggled on, the system would instead try to find the metric name and unit for each sensor from a specific row in your data source, allowing for different metrics to be defined within the same batch operation.

III. Column 3 - Since the toggle above is off, this section acts as the template for all custom sensors in the batch.

i. Metric - Enter the name of the measurement here (e.g., "Pressure," "Voltage"). This name will be used for every sensor created.

ii. Unit - Enter the unit of measurement (e.g., "PSI," "Volts"). This unit will be used for every sensor created.

 

Alarms

Alarms can be configured to let you know if issues occur during the import. 

  • Success Rate - An alarm will be raised on the import if a some rows of the import file file fail to import. The dropdown allows you to select a success threshold of some, most, or all rows parsed. 
  • Maximum Timeout - It is used to set an alarm if a process takes too long to complete. You can select a time duration (e.g., 1 hour, 6 hours), and an alarm would be triggered if the process exceeds that duration. 

9. Save and import - Once the configuration of the Manual Import profile is complete, the "Save and import" button, placed at top right corner of screen, will turn blue, indicating that the Import can be completed. Clicking "Save and import" will begin the import process.

 

Measurand SAA Import configuration

After selecting the Measurand ShapeArray option for your File Import you can either drag and drop the relevant SAA file or click "upload" and select the file from a file browser. 

Once that is done you will only need to set the name of the import, timezone, timestamp format and then define which sensor you intend to assign this import to. You can assign it to an already existing sensor or a new one. 

Selecting device

Here you can either select an existing device or create a new one for your import. 

 

Alarms

Alarms can be configured to let you know if issues occur during the import. 

  • Success Rate - An alarm will be raised on the import if a some rows of the import file file fail to import. The dropdown allows you to select a success threshold of some, most, or all rows parsed. 
  • Maximum Timeout - It is used to set an alarm if a process takes too long to complete. You can select a time duration (e.g., 1 hour, 6 hours), and an alarm would be triggered if the process exceeds that duration. 

Saving and Importing

Once you have everything correctly configured you can import the file.

If you need to configure the SAA any further then you can do so via the Explorer module. You can set cyclical adjustment, anti-rotation, convergence mode, reference date, configure the depth and length of each sensor, configure which end is fixed and if it is horizontal or vertical.

DGSI Manual Inclinometer Import configuration

After selecting the DGSI Manual Inclinometer option for your File Import you can either drag and drop the relevant SAA file or click "upload" and select the file from a file browser. 

Once that is done you will only need to set the name of the import, timezone, timestamp format and then define which sensor you intend to assign this import to. You can assign it to an already existing sensor or a new one. 

Automatically or manually importing data

If you want to import the data for your File Import automatically via FTP or to upload it manually please navigate to the Automatic and Manual File Import knowledge article.