Overview

Data Loading Descriptors (DLDs) are files that are stored in the Linear Analytics Services (LAS) database and which provide Linear Analytics Services with instructions on how to import various formats of data. You will never need to edit a DLD file directly; instead you use the "Load Data" panel on the Linear Analytics Services Web Interface to create and modify them. The DLD files you create and modify in this manner are automatically saved in the Linear Analytics Services database, although you can use Linear Analytics Services's "Database Files" tool to extract them from the database and save a local version of the file for back-up purposes or to copy a DLD to a different Linear Analytics Services server.

The most common use of DLDs is to reference them in a Linear Data Analysis (LDA) script that loads data. The DLD is referenced in the script in order to tell the script how to load the data. However, it is also possible to use a DLD directly from the "Load Data" panel on the Linear Analytics Services Web Interface in order to load a data file immediately without using a script. While this latter usage is possible, it is not recommended, since it is easy to accidentally change the DLD while doing this and any mistakes that are made when using a DLD directly from the "Load Data" panel are difficult to undo. You are advised to use the "Load Data" panel only to create and edit DLDs, and to always use LDA scripts to perform the actual loading of data.

Types of DLD

DLD files are used for importing the following data types into Linear Analytics Services:

• Event
• Asset
• Track
• Reference Line
• Marker
• Survey
• Design Curve
• Image Survey

Each of these types of DLD has its own parameters based on the idiosyncrasies of the data type it is importing, but they are all created or edited in the same way, and this page describes that process.

The process of creating or editing a DLD assumes that you have navigated to the Linear Analytics Services Web Interface and selected the Load Data page.

Loading an existing DLD

At the top of the Load Data page there is a section for loading an existing DLD in order to modify it or to use it to load a data file. Pressing the "Select" button will show a pop-up dialog containing a list of all the DLDs that are currently in the database, and selecting one from that dialog will populate the "DLD Name" field. You can then press "Load" to load the DLD into the page for editing or for use.

Creating a New DLD

DLDs can be set up to load data from either a file (a character-separated text file) or from an Oracle or Microsoft SQL Server database table or view.

The first tab on the creation panel is for creating a DLD based on a text file. The settings in this panel are as follows:

• Data Format Type - Use this drop-down to select which type of data this DLD is for.
• Data File - Use the "Select" button to open a dialog allowing you to select a data file to use as a template for the DLD. The file must be on the Linear Analytics Services server in your "User Files" area. If the file you wish to use as a template for the DLD is on your local machine rather than the Linear Analytics Services server then you can use the "Upload" button to upload it. In either case, the file name will then appear in the "Data File" field. This file does not have to actually be imported into Linear Analytics Services at this time. It is only needed as a template for the file format so that the DLD editor can read the columns that it contains and know what columns to therefore expect data files in this format to contain. Once the DLD has been created, the file does not need to be kept in the "User Files" area and can be safely moved or deleted without breaking the DLD. However, if you wish to edit the DLD at a later date you will need to re-upload the file or upload another file in the same format for the DLD to use as a template.
• Character Set - If your template file uses an unusual character set, you can select it from this drop down. For the majority of text files in ASCII or Unicode this setting can be left on "UTF-8", and you should leave it on this value if you do not know what character set your file uses.
• Delimiter - This setting tells Linear Analytics Services what character is used to separate columns in the data file.
• Header Row Exists - This tells Linear Analytics Services whether the first line of the file is a header row containing the names of the columns in the file, or whether the file has no header row and data starts immediately on the first row of the file.
• Comment Character - Many data files will have comment lines in them containing file metadata or other information which should not be interpreted as lines of data, and these lines often begin with a distinctive character to identify them. Linear Analytics Services will treat any line that begins with the character set in this setting as such a comment line and skip it when importing data.
• Quote Character - When data files are produced, sometimes one or more of the columns contains data in plain text format and this plain text can contain the character that is normally used as a column separator. It is normal for the software that writes data files to deal with this situation by putting a free text field in quotes - sometimes this is only done if the free text field contains a column separator character, and other times it is done with every free text field. This setting tells Linear Analytics Services how to recognise when this is the case. If a column in the file is contained in the quote characters selected in this setting then Linear Analytics Services will assume that these quote characters are not part of the data within the column and will strip them out. Additionally, if there are column separators between the opening and closing quote they will be interpreted as being part of the column's text rather than as a delimiter between columns.
• Date Format - This setting allows you to specify what format date fields within the file will be in, so that Linear Analytics Services can correctly interpret the text in those fields as dates. Date fields differ from timestamp fields (see below) in that they only contain information about a date, not about a particular time within that date. If a date field is used for information that is actually a timestamp then the time portion of the data will be lost and only the date will be preserved. Note that since only one date format can be provided here, if the data file does contain multiple date columns they must all have the same format. Hovering your mouse over the '?' at the end of this field will display pop-up help about the codes used in this setting to represent different parts of a date.
• Timestamp Format - This setting allows you to specify what format timestamp fields within the file will be in, so that Linear Analytics Services can correctly interpret the text in those fields as timestamps. Timestamp fields differ from date fields (see above) in that they include information about an exact time as well as information about a date. If a timestamp  field is used for data that is actually only a date then a time of midnight will be added to the date in order to turn it into a full timestamp. Note that since only one timestamp format can be provided here, if the data file does contain multiple timestamp columns they must all have the same format. Hovering your mouse over the '?' at the end of this field will display pop-up help about the codes used in this setting to represent different parts of a timestamp.
• Time Zone - This setting lets you define which time zone the timestamps in the file are from. If this is different from the time zone to which the Linear Analytics Services server is set then imported timestamps will be adjusted to account for the time difference between the two. If this setting is left as "Default" then any timestamps in imported files will be assumed to be in the same time zone as the Linear Analytics Services server and no adjustment will take place.

The second tab on the creation panel is for creating a DLD based on a database view or table. The settings on this panel are:

• Data Format Type - Use this drop-down to select which type of data this DLD is for.
• Database Type - Use this drop-down to select whether you are connecting to an Oracle database or a Microsoft SQL Server database.
• Server Name - This setting tells Linear Analytics Services the name of the database server to which it should try to connect.
• Port Number - This setting tells Linear Analytics Services what port number to use to connect to the database server. When you select the type of database you wish to connect to, this value will be automatically set to the default port for that database type, and you won't need to alter it unless your database has been set up to use a different port number.
• Database Name - (Microsoft SQL Server only) This setting lets you specify the name of the database that you wish to use on the database server.
• Service Name - (Oracle only) This setting lets you specify the service name of the database that you wish to use on the database server.
• User Name - This setting lets you specify which database user account you wish Linear Analytics Services to use to connect to the database. The database user account must have read permission on the table or view containing the data.
• Password - This setting lets you specify the password that accompanies the database user account you wish Linear Analytics Services to use to connect to the database. Clicking the button at the end of the password field toggles whether or not the characters you type should be visible. If you are having trouble typing the password accurately without being able to see what you're typing you can turn this on, but obviously you should not do so if there is the possibility of other people viewing your screen either physically or electronically.
• Table/View Name - This setting lets you specify the name of the table or view within the database that contains the data to be imported.

Unlike creating a DLD from a file, where the file is only there as a template and is not used when you import data using an LDA script, when you create a DLD from a database it will always use the same database connection and table/view to import data from whenever you import using the DLD.

Modifying a DLD

When you create a DLD or load an existing DLD, you are taken to the DLD editing page where you can set up the details of how the DLD loads data. Some parts of this page are common to all types of DLD and are described here. Parts of the DLD editing process that are specific to particular types of DLD are described in the pages for creating those types of DLD.

At the top of the DLD editing page is a summary of the settings initially used to create the DLD; either file settings (as shown above) or database connection settings. This summary is read-only in that you can't directly edit these settings from this page. If you need to edit these settings (for example if the file that you are using as a template is no longer there and you need to specify a new template file), click the "Update" button. This will pop up a dialog containing the DLD creation settings and allow you to modify any of them.

Data Specific Settings

The majority of the controls on the DLD editing page are specific to the type of DLD that is being edited. See the relevant page for the DLD type:

• Event - [[Creating a DLD for Event Data]]
• Asset - [[Creating a DLD for Asset Data]]
• Reference Line - [[Creating a DLD for Reference Line Data]]
• Marker - [[Creating a DLD for Marker Data]]
• Survey - [[Creating a DLD for Stream Data]]
• Design Curve - [[Creating a DLD for Design Curve Data]]
• Image Survey - [[Creating a DLD for Image Data]]

Saving Your DLD and/or Importing Data Immediately

At the bottom of the DLD editing page are a number of controls for saving your DLD and/or loading data with it.

The "Log Level" setting is only used if you are importing data immediately. It tells Linear Analytics Services how much information to write to its logs about the import process. The four settings are:

• Information - all information is written to the log.
• Warnings - warnings and errors are written to the log, but miscellaneous information is not written.
• Errors - only errors are written to the log. Warnings are ignored.
• Fatal - only errors that would stop the import are written to the log. Lesser warnings (that might result in some data not being imported but which won't stop the whole import process) are ignored.

The "DLD Name" field is for you to supply the name of the DLD. The DLD is a file with the suffix '.dld', so that suffix will be added to the name you supply if not already present. This file name is the name under which the DLD can be found in the "Database Files" tool in Linear Analytics Services, as well as the name under which it must be referenced if it is to be used in an LDA script. If you change the name of a DLD and re-save it, this will create a new copy of the DLD with the new name and leave the old version intact. If you specify a name that is already the name of an existing DLD then the existing DLD will be overwritten with this one.

The "Save" button will save the DLD to the Linear Analytics Services database under then name you have specified above, and the "Import" button will use the DLD to import the data in the template file or the database (depending on the type of DLD you are editing). The "Save & Import" button does both these, one after the other. The saving is done first, so if the import fails, the DLD will have already been saved. Again, while it is possible to load data into Linear Analytics Services by opening its DLD for editing and using these buttons, it is not recommended that you do this on a production server, since it is easy to accidentally change the DLD in the process and difficult to undo any mistakes made when loading data in this way. You are advised to load data only with LDA scripts.

The "Reload" button will wipe all changes that you have made to the DLD and re-load the version that is stored in the Linear Analytics Services database (if there is one).

The "Exit DLD" button will stop editing the DLD and return you to the DLD creation page. If you have unsaved changes to your DLD then you will be prompted to save them first.