File upload guide

This guide walks you through the process of uploading data to WeSIS.

Warnings: Currently, we are only saving monadic data. We are still working on processing dyadic data. You can test the uploading validations for dyadic files, but the data won't be saved in the database. Furthermore, some country codes are not accepted yet.

Please use the "Discussion" tab of this page to discuss further details.

Before you start

When you’ve decided that your data is finally ready to be uploaded to WeSIS please make sure that you have:

1. Created an indicator page on WeSISpedia. You should use the following template. Here is an example indicator page to give you an idea of what the page structure should look like. Please make sure that all relevant fields in the info box are filled out, since this is important for data validation.
2. Added all new indicators to the appropriate topic index pages on WeSISpedia. WeSIS only recognizes indicators that exist on these pages and have the mandatory columns filled out.
3. Formatted your files according to the provided templates as described in File formats. Please note that the system accepts .csv,.xls and .xlsx. Though uploading .xls files is not recommended.
4. Created a user account in WeSIS. Even if you participated in a previous testing session, this is the first time that WeSIS is online. Therefore, everyone needs to create a new account.

Steps Outline

In this introduction, you'll see a quick demonstration of the upload process. (gif or video to be added). Or you can read the text below.

The upload process consists of two main steps.

Step 1: Upload dataset files

At this step you can select or simply drag-n-drop your file into the upload area and press "SAVE".

Note: Big files may take a while to be validated (~2-5 minutes).

Step 2: Preview and Data Validation

At this step you are presented with a file overview, where you can:

1. Change the file format to be used between "monadic" or "dyadic" (however, as mentioned above, we do not recognize the dyadic one yet)
2. See all recognized mandatory columns, technical indicator names, optional columns, countries and year values fetched from the file by hovering over the appropriate "See full list" fields.
3. Go back to the WeSIS home page by clicking the "Back to homepage" button.
4. Access this upload guide by using the "Upload Guide" link on the bottom right.
5. Open the file preview page, which shows the uploaded data with color-coded cells for different types of validation errors if present.
6. Upload a new updated file to the system by clicking the "Reupload File" button, which brings the user back to step 1.

On the right, you can see the validation logs output and have the option to download either your .csv file updated with a special column indicating the rows with errors or the logs themselves as a .txt file.

If the file passed all validation checks, the Parsing Logs box will be empty. Click on the "Upload" button to complete the upload process.

Uploading the validated data to the database may take a few hours, especially with files over 1000 rows. Please check tomorrow if the indicator values are visible in WeSIS. If they are not, contact an A01 project member.

Error logs

At the moment the system performs the following types of error checks:

1. Missing_columns - checks for mandatory columns missing in the uploaded file.
2. Multiple_triples - validates that there is no more than one entry for each (cow_code, year, technical_variable_name) triple.
3. Invalid_datatype - checks whether cells are of type desribed in File formats section, i.e. Numeric, String, Binary, Datetime or other.
4. Unrecognized_values - checks whether the following column values: country, cow_code, technical_indicator_name, scale exist in database. If not, the user should update the appropriate WesSISPedia page.
5. TechName/Scale_mismatch - checks whether the scale documented in WeSISPedia for each technical indicator agrees with the scale in the file.
6. Scale/Value_mismatch - checks whether the scale chosen in the file agrees with the scale of the actual value.
7. CowCode/CountryName_mismatch - checks whether the country_name and cow_code pairs agrees with country_name and cow_code pairs documented in WeSISPedia.

The logic used to validate data is shown in the flowchart below.

File preview

There are two options when previewing the data. Each can be toggled on or off by pressing the appropriate buttons located above the table on the left.

• "Show/Hide All Rows" allows the user to show/hide all rows from the uploaded file. By default, the table in the data preview shows only the rows with errors.
• "Show/Hide Optional Columns" allows the user to show/hide non-mandatory columns if present.

Please keep in mind that the row numbering starts on the first row with data. Therefore, the numbering may not perfectly match the numbering of your file.