WeSISpedia - User contributions [en]

Social structure

2024-06-11T09:11:11Z

Hsalzmann:

{|class="wikitable sortable" border="1"
|-
! Indicator Name
! Subcategory I
! Subcategory II
! Subcategory III
! Technical name
! Scale
! Short description
! Data origin*
|-
| [[Employment to population ratio, 15+, male (national estimate)]]
| Labour market participation
| Employment
| Rate
| socstr_emp_pop_15m
| Metric
| Measures the employment ratio of a country’s male population aged 15 or older (WBG).
|
|-
| [[Employment to population ratio, 15+, female (national estimate)]]
| Labour market participation
| Employment
| Rate
| socstr_emp_pop_15f
| Metric
| Measures the employment ratio of a country’s female population aged 15 or older (WBG).
|
|-
| [[Employment to population ratio, 15+, total (national estimate)]]
| Labour market participation
| Employment
| Rate
| socstr_emp_pop_15t
| Metric
| Measures the employment ratio of a country’s population aged 15 or older (WBG).
|
|-
| [[Employment to population ratio, 15-24, male (national estimate)]]
| Labour market participation
| Employment
| Rate
| socstr_emp_pop_1524m
| Metric
| Measures the employment ratio of a country’s male population aged 15 to 24 (WBG).
|
|-
| [[Employment to population ratio, 15-24, female (national estimate)]]
| Labour market participation
| Employment
| Rate
| socstr_emp_pop_1524f
| Metric
| Measures the employment ratio of a country’s female population aged 15 to 24 (WBG).
|
|-
| [[Employment to population ratio, 15-24, total (national estimate)]]
| Labour market participation
| Employment
| Rate
| socstr_emp_pop_1524t
| Metric
| Measures the employment ratio of a country’s population aged 15 to 24 (WBG).
|
|-
| [[Labor force participation rate, 15+, male (national estimate)]]
| Labour market participation
| Labour force participation
| Rate
| socstr_lfp_pop_15m
| Metric
| Measures the economically active proportion of a country's male population aged 15 or older (WBG).
|
|-
| [[Labor force participation rate, 15+, female (national estimate)]]
| Labour market participation
| Labour force participation
| Rate
| socstr_lfp_pop_15f
| Metric
| Measures the economically active proportion of a country's female population aged 15 or older (WBG).
|
|-
| [[Labor force participation rate, 15+, total (national estimate)]]
| Labour market participation
| Labour force participation
| Rate
| socstr_lfp_pop_15t
| Metric
| Measures the economically active proportion of a country's population aged 15 or older (WBG).
|
|-
| [[Labor force with advanced education/total working-age population]]
| Labour market participation
| Education
| Rate
| socstr_lfpedu3_wpop_t
| Metric
| Measures the economically active proportion of a country's population aged 15 or older and gained an advanced level of education (WBG).
|
|-
| [[Labor force with advanced education/female working-age population]]
| Labour market participation
| Education
| Rate
| socstr_lfpedu3_wpop_f
| Metric
| Measures the economically active proportion of a country's female population aged 15 or older and gained an advanced level of education (WBG).
|
|-
| [[Labor force with advanced education/male working-age population)]]
| Labour market participation
| Education
| Rate
| socstr_lfpedu3_wpop_m
| Metric
| Measures the economically active proportion of a country's male population aged 15 or older and gained an advanced level of education (WBG).
|
|-
| [[Labor force with intermediate education/total working-age population]]
| Labour market participation
| Education
| Rate
| socstr_lfpedu2_wpop_t
| Metric
| Measures the economically active proportion of a country's population aged 15 or older and gained an intermediate level of education (WBG).
|
|-
| [[Labor force with intermediate education/female working-age population]]
| Labour market participation
| Education
| Rate
| socstr_lfpedu2_wpop_f
| Metric
| Measures the economically active proportion of a country's female population aged 15 or older and gained an intermediate level of education (WBG).
|
|-
| [[Labor force with intermediate education/male working-age population)]]
| Labour market participation
| Education
| Rate
| socstr_lfpedu2_wpop_15m
| Metric
| Measures the economically active proportion of a country's male population aged 15 or older and gained an intermediate level of education (WBG).
|
|-
| [[Labor force with basic education/total working-age population]]
| Labour market participation
| Education
| Rate
| socstr_lfpedu1_wpop_t
| Metric
| Measures the economically active proportion of a country's population aged 15 or older and gained a basic level of education (WBG).
|
|-
| [[Labor force with basic education/female working-age population]]
| Labour market participation
| Education
| Rate
| socstr_lfpedu1_wpop_f
| Metric
| Measures the economically active proportion of a country's female population aged 15 or older and gained a basic level of education (WBG).
|
|-
| [[Labor force with basic education/male working-age population]]
| Labour market participation
| Education
| Rate
| socstr_lfpedu1_wpop_15m
| Metric
| Measures the economically active proportion of a country's male population aged 15 or older and gained a basic level of education (WBG).
|
|-
| [[Atkinson Coefficient (epsilon=0.5)]]
| Economic inequality
| Inequality measure
| Index value
| socstr_atkincoef_eps05
| Metric
| Measures income inequality of a country under considerations of further parameters like society's aversion of income inequality (LIS).
|
|-
| [[Atkinson Coefficient (epsilon=1)]]
| Economic inequality
| Inequality measure
| Index value
| socstr_atkincoef_eps1
| Metric
| Measures income inequality of a country under considerations of further parameters like society's aversion of income inequality (LIS).
|
|-
| [[Gini Coefficient (LIS)]]
| Economic inequality
| Inequality measure
| Index value
| socstr_gini_coef_lis
| Metric
| Measures income inequality of a country's population and ranges between absolute income inequality and equal distribution of income. This Gini Coeffizient is based on calculation from the Luxembourg Income Study Database (LIS).
|
|-
| [[Gini (at disposable income post taxes & transfers)]]
| Economic inequality
| Inequality measure
| Index value
| socstr_gini_disinc_oecd
| Metric
| Measures income inequality of disposable income of a country's population after taxes and other transfers. Ranges between absolute income inequality and equal distribution of income. This Gini Coeffizient is based on calculation from the Organisation for Economic Co-operation and Development (OECD).
|
|-
| [[Income inequality: S80/S20 disposable income quintile share]]
| Economic inequality
| Inequality measure
| Index value
| socstr_s80s20_disinc_oecd
| Metric
| Measures income inequality as ratio of the disposable income decile S80/S20 (OECD).
|
|-
| [[Income inequality: P90/P10 disposable income decile ratio]]
| Economic inequality
| Inequality measure
| Index value
| socstr_p90p10_disinc_oecd
| Metric
| Measures income inequality as ratio of the disposable income decile P90/P10 (OECD).
|
|-
| [[Income inequality: P90/P50 disposable income decile ratio]]
| Economic inequality
| Inequality measure
| Index value
| socstr_p90p50_disinc_oecd
| Metric
| Measures income inequality as ratio of the disposable income decile P90/P50 (OECD).
|
|-
| [[Income inequality: P50/P10 disposable income decile ratio]]
| Economic inequality
| Inequality measure
| Index value
| socstr_p50p10_disinc_oecd
| Metric
| Measures income inequality as ratio of the disposable income decile P50/P10 (OECD).
|
|-
| [[GINI index (World Bank estimate)]]
| Economic inequality
| Inequality measure
| Index value
| socstr_gini_ndx_wb
| Metric
| Measures income inequality of a country's population and ranges between absolute income inequality and equal distribution of income. This Gini Coeffizient is based on calculation from the World Bank (WB).
|
|-
| [[Gini Coefficient (WIID)]]
| Economic inequality
| Inequality measure
| Index value
| socstr_gini_coef_wiid4
| Metric
| Measures income inequality of a country's population and ranges between absolute income inequality and equal distribution of income. This Gini Coeffizient is based on calculation from the World Institute for Development Economics Research (UNU-WIDER / WIID).
|
|-
| [[Human Development Index]]
| Human Development
| Development indicator
| Index value
| socstr_hdi_undp
| Metric
| Measures average achievement of a country in various dimensions of human development like health, education and living standard, but not inequality, poverty or empowerment (UNDP).
|
|-
| [[Relative poverty rates: Entire population]]
| Economic inequality
| Poverty measure
| Rate
| socstr_repovratetpop_oecd
| Metric
| Measures relative poverty of the entire population of a country (OECD).
|
|-
| [[Relative poverty rates: Children (age 0-17)]]
| Economic inequality
| Poverty measure
| Rate
| socstr_repovratechild_oecd
| Metric
| Measures relative poverty of children aged 0 up to 17 of a country (OECD).
|
|-
| [[Relative poverty rates: Working-age population (age 18-65)]]
| Economic inequality
| Poverty measure
| Rate
| socstr_repovratewap_oecd
| Metric
| Measures relative poverty of the working age population aged 18 up to 65of a country (OECD).
|
|-
| [[Relative poverty rates: Retirement-age population (over 65)]]
| Economic inequality
| Poverty measure
| Rate
| socstr_repovratera_oecd
| Metric
| Measures relative poverty of the retirement-age population aged 65 and older of a country (OECD).
|
|-
| [[Poverty gap entire population]]
| Economic inequality
| Poverty measure
| Percentage
| socstr_povgaptpop_oecd
| Metric
| Measures the poverty gap of a country's entire population (OECD).
|
|-
| [[Poverty gap at dollar 1.90 a day (2011 PPP) (%)]]
| Economic inequality
| Poverty measure
| Percentage
| socstr_povgap190_11_wb
| Metric
| Measures poverty with a poverty gap set at 1.90 dollars a day which reflects the depth of poverty and its incidence (WB).
|
|-
| [[Poverty gap at dollar 3.20 a day (2011 PPP) (%)]]
| Economic inequality
| Poverty measure
| Percentage
| socstr_povgap320_11_wb
| Metric
| Measures poverty with a poverty gap set at 3.20 dollars a day which reflects the depth of poverty and its incidence (WB).
|
|-
| [[Urban poverty headcount ratio at national poverty lines (% of urban pop.)]]
| Economic inequality
| Poverty measure
| Rate
| socstr_urbpovratio_wb
| Metric
| Measures the percentage of a country's urban population living in poverty (WB).
|
|-
| [[Poverty gap at national poverty lines (%)]]
| Economic inequality
| Poverty measure
| Percentage
| socstr_povgapnat_wb
| Metric
| Measures the percentage of a country's poor population below the mean shortfall to the poor population above the mean shortfall (WB).
|
|-
| [[Rural poverty headcount ratio at national poverty lines (% of rural pop.)]]
| Economic inequality
| Poverty measure
| Percentage
| socstr_rurpovratio_wb
| Metric
| Measures the percentage of a country's rural population living in poverty (WB).
|
|-
| [[Gender Inequality Index]]
| Social inequality
| Inequality measure
| Index value
| socstr_gdi_undp
| Metric
| Measures inequality in gender based on the three dimensions reproductive health, empowerment, and labour market (UNDP).
|
|-
| [[Population, total]]
|
|
|
| socstr_pop_tot_unctad
| Metric
| Absolute value of population based on UNCTAD statistics
|
|-
| [[Total male population]]
| Population
|
| Thousands
| socstr_pop_totma_pp
| Metric
|
|
|-
| [[Total female population]]
| Population
|
| Thousands
| socstr_pop_totfe_pp
| Metric
|
|
|-
| [[Total population, both sexes]]
| Population
|
| Thousands
| socstr_pop_tot_pp
| Metric
|
|
|-
| [[Population per square kilometre]]
| Population
|
| Thousands
| socstr_pop_dens_pp
| Metric
|
|
|-
| [[Male population ages 0-14]]
| Population
|
| Thousands
| socstr_pop_0014ma_pp
| Metric
|
|
|-
| [[Female population ages 0-14]]
| Population
|
| Thousands
| socstr_pop_0014fe_pp
| Metric
|
|
|-
| [[Total population ages 0-14]]
| Population
|
| Thousands
| socstr_pop_0014to_pp
| Metric
|
|
|-
| [[Male population ages 0-19]]
| Population
|
| Thousands
| socstr_pop_0019ma_pp
| Metric
|
|
|-
| [[Female population ages 0-19]]
| Population
|
| Thousands
| socstr_pop_0019fe_pp
| Metric
|
|
|-
| [[Total population ages 0-19]]
| Population
|
| Thousands
| socstr_pop_0019to_pp
| Metric
|
|
|-
| [[Male population ages 15-64]]
| Population
|
| Thousands
| socstr_pop_1564ma_pp
| Metric
|
|
|-
| [[Female population ages 15-64]]
| Population
|
| Thousands
| socstr_pop_1564fe_pp
| Metric
|
|
|-
| [[Total population ages 15-64]]
| Population
|
| Thousands
| socstr_pop_1564to_pp
| Metric
|
|
|-
| [[Male population ages 0-64]]
| Population
|
| Thousands
| socstr_pop_0064ma_pp
| Metric
|
|
|-
| [[Female population ages 0-64]]
| Population
|
| Thousands
| socstr_pop_0064fe_pp
| Metric
|
|
|-
| [[Total population ages 0-64]]
| Population
|
| Thousands
| socstr_pop_0064to_pp
| Metric
|
|
|-
| [[Male population ages 60 and above]]
| Population
|
| Thousands
| socstr_pop_60upma_pp
| Metric
|
|
|-
| [[Female population ages 60 and above]]
| Population
|
| Thousands
| socstr_pop_60upfe_pp
| Metric
|
|
|-
| [[Total population ages 60 and above]]
| Population
|
| Thousands
| socstr_pop_60upto_pp
| Metric
|
|
|-
| [[Male population ages 65 and above]]
| Population
|
| Thousands
| socstr_pop_65upma_pp
| Metric
|
|
|-
| [[Female population ages 65 and above]]
| Population
|
| Thousands
| socstr_pop_65upfe_pp
| Metric
|
|
|-
| [[Total population ages 65 and above]]
| Population
|
| Thousands
| socstr_pop_65upto_pp
| Metric
|
|
|-
| [[Total fertility rate]]
| Population
|
| Children per woman
| socstr_pop_tfr_gp
| Metric
|
|
|-
| [[Life expectancy at birth]]
| Population
|
| Years
| socstr_pop_lex_gp
| Metric
|
|
|-
| [[Global Gender Gap Index]]
| Social inequality
| Inequality measure
| Index value
| socstr_si_im_gggi
| Metric
| The index benchmarks national gender gaps on economic, education, health and political criteria (WEF).
|
|-
| [[Gender Wage Gap]]
| Social inequality
| Inequality measure
| Gap
| socstr__si_im_gwg
| Metric
| Difference between median earnings of men and women relative to median earnings of men (OECD).
|
|-
| [[Maternal Employment Rate, partnered mothers]]
| Labour market participation
| Employment
| Rate
| socstr_lm_ep_merpm
| Metric
| Employment rates (%) for partnered mothers (15-64 year olds) with at least one child under 15 (OECD).
|
|-
| [[Maternal Employment Rate, sole-parent mothers]]
| Labour market participation
| Employment
| Rate
| socstr_lm_ep_mersm
| Metric
| Employment rates (%) for sole-parent mothers (15-64 year olds) with at least one child under 15 (OECD).
|
|-
| [[Children in employment, total ]]
| Labour market participation
| Employment
| Percentage
| socstr_lm_ep_cept
| Metric
| This indicator measures children involved in economic activity aged 7 – 14 and not attending school (Understanding Children’s Work).
|
|-
| [[Children in employment, female ]]
| Labour market participation
| Employment
| Percentage
| socstr_lm_ep_cepf
| Metric
| This indicator measures female children involved in economic activity aged 7 – 14 and not attending school (Understanding Children’s Work).
|
|-
| [[Children in employment, male ]]
| Labour market participation
| Employment
| Percentage
| socstr_lm_ep_cepm
| Metric
| This indicator measures male children involved in economic activity aged 7 – 14 and not attending school (Understanding Children’s Work).
|
|-
| [[Test gini prediction]]
| Economic inequality
| Inequality measure
| Index value
| socstr_test_gini_prediction
| Metric
| Predictions of income gini values based on other indicators within WeSIS.
| Original CRC Data
|}

Data quality

2024-06-11T09:10:59Z

Hsalzmann:

Data quality in the social sciences is often discussed under the headings of validity and reliability when talking about concepts, measurement and data.
However, few discuss transparency and data quality, and there is no consistent understanding what "data quality" means in the social sciences.
This page describes the process, the rationale and the measures we undertook for improving data quality in WeSIS.

__TOC__

== Introduction ==
In their famous book "Designing Social Inquiry", King et al. (1994, 24) set out five guidelines for "improving data quality" encompassing

# valid measures,
# reliable data collections,
# replicable analyses,
# a thorough documentation of the data generating process, and
# to "collect data on as many of [a theory’s] observable implications as possible".

Nowadays, much more data is available posing new challenges, though; counterfeit data, typos, and unintended mistakes in the data collection, for example, pose challenges beyond invalid and unreliable measures.
The structured documentation of each indicator (including meta data) in WeSISpedia, common coding rules and formatting standards alongside the resources provided in WeSIS, for example, [https://wesis.org/datasets public data sets], [https://wesis.org/tools/public_scripts community notebooks] or an indicator's version control, directly speak to King et al.'s suggestions.
Still, we also opted for a broader concept based on data quality definitions commonly used in information systems design (cf. Pipino et al. 2002). ''Concise'' and ''consistent representation'' or "''free-of-errorness''", for example, are taken care of by the validation check performed when uploading data to WeSIS (for more details see the [[File_upload_guide|file upload guide]]).

While such automated checks improve, among others, the compatibility with other social science data collections and thus increase the "I" and "R" in the FAIR concept of research data (cf. Wilkinson et al. 2016) -- namely "interoperability" and "re-usability" -- there are more vague aspects of data quality like believability, reputation or objectivity that go beyond the idea of statistical confidence and uncertainty measures.
Seldom adressed, or even made explicit, in social science research, they still are at the core of many scientist's common-sensical understanding of "data quality", particularly given their domain-specific knowledge.

Initial discussions in the CRC on implementing a "traffic light system" for rating "data accuracy" brought out the complexity, or rather difficulty, of designing a scheme that is abstract and intuitive enough to share the knowlodge ''about'' the data and immediately reveal this ''meta'' data to external users while at the same time being simple to code and broad enough to cover all, or at least "typical" instances that occured or may occur in the "data generation process".
At the same time, there is a sparsity of attempts in the social sciences to rate data, not to speak of a "gold standard" of a framework.

Further workshops therefore focused on a proposal for rating the quality of data in WeSIS rooted in a "provenance approach" (see e.g., Prat and Madnick 2007 or Buneman et al. 2001) which puts the origin of the data in the spotlight.
The proposal entailed rating three dimensions (trustworthiness of a source, reasonableness of data, and confidence in data), each connected to a suggested rating scale drawn from other disciplines e.g., the likelihood scale from the IPBES "Guide on the production of assessment" (IPBES 2018).
The main outcome of the discussion was that the first two dimensions were quite uncontroversial although the suggested scales needed refinement and adoption to the peculiarities of the CRC's data collections.
However, the discussion also showed that despite a common understanding of the value of such ratings, social science is still marked by a resistance to "judge" the data given the extra work necessary to define a feasible scale and wordings ''and'' apply it, and concerns about "devaluing one's own work" or deviations and bias across people and projects were raised.

Two subsequent workshops underlined the difficulty of balancing the commitment for implementing a rating scheme vs. prospect of creating an added value vs. (additional) work load required to provide and code the necessary information.
Ultimately, an agreement was reached that goes beyond the "typical" social science concepts of validity and reliability entailing (1) to label any indicator on the indicator level whether it was "created" by domain experts of the CRC, and (2) to code notable concerns at the data point/value level.

== Dimensions of data quality in WeSIS ==

Against this background, "data quality" in WeSIS encompasses three main elements:

'''1. Automated validation checks'''

''Concise'' and ''consistent representation'' and "''free-of-errorness''" are taken care of by the validation check performed when uploading data to WeSIS (for more details see the [[File_upload_guide|file upload guide]]).
The check not only requires the data providers to comply to the agreed [[coding rules]] and [[File_formats|file formatting standards]] but also checks for "common" mistakes like typos, country code--name or value--scale mismatches.

'''2. Data origin'''

Beyond the structured documentation of each indicator detailing the sources, each indicator is labeled according to one of three categories referring to the origin of the data on the most abstract level.

{| class="wikitable"
|-
! Category
! Meaning
|-
| Original CRC Data
| Data gathered, compiled and coded by CRC researchers, e.g., qualitative judgments, text codings or annotations.
|-
| CRC Compiled Data
| Data gathered, harmonized and/or extended by CRC researchers, i.e., merely statistical data aggregated via reproducible workflows (scripts) incl. imputations.
|-
| 3rd Party Data
| Data taken "as is" without further treatments from third party sources, except for matching the source's country codes to the WeSIS scheme and adding the necessary meta data required for upload to WeSIS.
|}

As the labels refer to an entire indicator, they are input via the mandatory column "data origin" of each indicator ''table'' page (e.g., [[Old age and survivors]]).
In WeSIS, the information is shown as a label (or sticker) on each indicator page right below the variable name.

'''3. Plausibility concerns'''

The workshops made clear that any numerical value like a likelihood of a value would not fit the heterogeneity of concerns encountered in the CRC's data collections.
Instead, the rating captures the concerns researchers may have regarding the consistency or plausibility of a data value, i.e., at the data point level, based on their domain knowledge.
The guiding question and corresponding nominal response categories are:

''Q: "Are there notable concerns about the plausibility of this data point?"''

{| class="wikitable"
|-
! Response
! Meaning
|-
| No
| (The default)
|-
| Yes, from a time-series perspective
| Concerns refer to implausible values like breaks or jumps in a time-series or large deviations for which there is no obvious explanation.
|-
| Yes, from a cross-sectional perspective
| Concerns refer to implausible values when comparing the focal value to values/levels of other entities given the focal time point.
|-
| Yes, due to limited sources
| Concerns refer to values where there is limited information from uncertain or unsound sources.
|-
| Yes, due to contradicting sources
| Concerns refer to values where at least two sources exist but contain contradicting or otherwise mis-matching information.
|-
| Other
| Reasons not covered by the other response categories.
|}

If multiple concerns are raised for the focal value, only the most prevalent one is coded.
If a researcher chooses "Other" as the response, it is mandatory to provide an explanation in the [[Optional_column_names|optional column]] <code>comment</code> in the WeSIS upload file template.
As the concerns are expressed at the value level, they are input together with the actual data via the mandatory column <code>plausibility_concerns</code> of the [[File_formats|file upload templates]].
In WeSIS, the information is available in the data table of each indicator page.

== Technical implementation ==

To implement the agreed solution, with the release of WeSIS version 2.5 the upload workflow was modified.
After discussing the pros and cons of different variants (e.g., parsing from the indicator table in WeSISpedia, adding a mandatory column to the upload file template, or exploiting the very first line of the csv file), it was decided to grab the information for the label from the indicator ''table'' pages in WeSISpedia.
As these are also used for the validation check during the upload simply adding a column "data_origin" to the table of each policy field served the purpose.
This column is now mandatory, standardized, and of type "String" allowing only the above mentioned three categories for coding the origin of data, else nothing will be displayed in WeSIS.
The database (DB) structure was affected in that only one additional column called "data_origin" was added to the DB table "indicators".

For the plausibility concerns, the [[File_formats|file upload templates]] were appended to include a new column "plausibility_concerns".
This column is now mandatory, standardized, and of type "String" allowing only the above mentioned six categories for coding researcher's concerns.
The DB structure was affected in that one additional column called "plausibility_concerns" was added to the database tables "ds_values" and "ds_dyadic_values".

The [[File_formats_archive|first version]] of the file upload template in place until November 2023 already included two placeholders for a data quality "rating", namely the then mandatory columns "data_quality" and "data_quality_confidence" as placeholders for a "traffic light" or some other scheme.
While the codings remained undetermined at that time, some data providers made use of them inputting their own remarks, reasonings and quality judgements. To not lose this information,
* the information contained in the DB was kept,
* but both columns are now marked as deprecated;
* the validation check ignores whether the upload file (still) contains the old columns, or not, and treats them as [[Optional_column_names|optional column names]]; this way, the researchers are still able to update "their" information for data already existent before the above agreement was reached.

For existing data already in the DB, the plausibility rating is not available (and will not be unless the data is updated); until the codings will gradually become available with an indicator update, it was agreed that WeSIS will display a "Not Available" in the data table for the time being.

== References ==

* Buneman, Peter, Sanjeev Khanna, and Wang-Chiew Tan. 2001. ''Why and Where: A Characterization of Data Provenance''. University of Pennsylvania, Pennsylvania: Department of Computer & Information Science. Departmental Papers (CIS).
* Intergovernmental Science-Policy Platform on Biodiversity and Ecosystem Services (IPBES). 2018. ''The IPBES Guide on the production of assessments. Core version''. IPBES/6/INF/17. [https://www.ipbes.net/document-library-catalogue/ipbes6inf17 https://www.ipbes.net/document-library-catalogue/ipbes6inf17] (Accessed January 17, 2023)
* King, Gary, Robert O. Keohane, and Sidney Verba. 1994. ''Designing Social Inquiry: Scientific Inference in Qualitative Research''. Princeton: Princeton University Press.
* Pipino, Leo L., Yang W. Lee, and Richard Y. Wang. 2002. "Data quality assessment." ''Communications of the ACM'' 45 (4): 211–18. [https://doi.org/10.1145/505248.506010 https://doi.org/10.1145/505248.506010]
* Prat, Nicolas, and Stuart E. Madnick. 2007. ''Measuring Data Believability: A Provenance Approach''. Massachusetts Institute of Technology, Cambridge: Composite Information Systems Laboratory (CISL), Sloan School of Management.
* Wilkinson, Mark D., Michel Dumontier, I. J. J. Aalbersberg, Gabrielle Appleton, Myles Axton, Arie Baak, Niklas Blomberg, Jan-Willem Boiten, Luiz B. da Silva Santos, Philip E. Bourne, Jildau Bouwman, Anthony J. Brookes, Tim Clark, Mercè Crosas, Ingrid Dillo, Olivier Dumon, Scott Edmunds, Chris T. Evelo, Richard Finkers, Alejandra Gonzalez-Beltran, Alasdair J. G. Gray, Paul Groth, Carole Goble, Jeffrey S. Grethe, Jaap Heringa, Peter A. C. 't Hoen, Rob Hooft, Tobias Kuhn, Ruben Kok, Joost Kok, Scott J. Lusher, Maryann E. Martone, Albert Mons, Abel L. Packer, Bengt Persson, Philippe Rocca-Serra, Marco Roos, Rene van Schaik, Susanna-Assunta Sansone, Erik Schultes, Thierry Sengstag, Ted Slater, George Strawn, Morris A. Swertz, Mark Thompson, Johan van der Lei, Erik van Mulligen, Jan Velterop, Andra Waagmeester, Peter Wittenburg, Katherine Wolstencroft, Jun Zhao, and Barend Mons. 2016. "The FAIR Guiding Principles for scientific data management and stewardship." ''Scientific data'' 3: 160018. [https://doi.org/10.1038/sdata.2016.18 https://doi.org/10.1038/sdata.2016.18]

Social structure

2024-06-11T08:36:45Z

Hsalzmann:

File formats

2024-06-11T07:47:11Z

Hsalzmann:

This page describes the file formats suitable and accepted for data upload to WeSIS as of November 2023.

Two templates - one for monadic and one for dyadic data - are available to CRC researches in the CRC shared Nextcloud folder "WeSIS Phase 2 > Data Collection > Data Templates" (each as a csv- and a xlsx-file).
Both templates are approved and ready to be applied.

After a longer process, a conclusion was reached how to mark [[Data_quality|data quality]].
In addition, a new way of flagging [[File_upload_guide#Uploading_N/A_values|"missing values"]] was introduced.
Accordingly, the template was changed two include two new mandatory columns, namely <code>user_na_values</code> and <code>plausibility_concerns</code>.
The former columns <code>data_quality</code> and <code>data_quality_confidence</code> were marked as ''deprecated'' (they are now [[Optional_column_names|optional]] in order to retain old data in the database; and you can still access the documentation for the [[File_formats_archive|old template]]).

__TOC__

== General remarks ==

When preparing the upload file, the '''first row''' - and only the first row (!) - can be used for any kind of additional information (coder information, project notes, last updated etc.).
The line '''must''' start with the hashtag character '''#''' (as if it was a comment in R).

Many [[Coding_rules|coding rules]] have been agreed upon and shall be followed. Please be aware of these rules and regularly check the [[Coding_rules|site]] to see if further decisions have been added. These rules encompass among others country and IO names and codes, date time formats, technical variable names and other aspects that affect the data collection, ensure consistency and reduce error.

Both templates include
# mandatory and standardized columns for the data ''per se'',
# mandatory and standardized columns for metadata, and
# (unlimited) optional columns for additional information regarding the data.

Standards for optional column names will evolve applying the principle of "first come, first serve" that was agreed upon. If you want to use optional columns,
* check the page on [[Optional_column_names|optional column names]] if any other project already made a suggestion that fits your needs.
* If a name already exists and you find the description suitable to file your information under this heading, please apply the existing one.
* If there is no name that suits the information you want to store, create a new name and describe the purpose of the column on the [[Optional_column_names|page]] so others may use it later on.

== Mandatory columns for the monadic data template ==

{|class="wikitable sortable" border="1"
!Column name!!Column number!!Standardized?!!Type||Description!!Comments
|-
|cow_code||1||yes||Numeric||Country code according to the COW scheme||See the list of [[Country_codes|country and IO codes]] used for WeSIS and WeSISpedia.
|-
|country_name||2||yes||String||Country name||See the list of [[Country_codes|country and IO codes]] used for WeSIS and WeSISpedia.
|-
|year||3||yes||Numeric||The year ''value'' refers to||Note that a date, like election dates or the date of introduction of a policy, is a ''value''. ''year'', thus, may "double" the value.
|-
|technical_variable_name||4||partly||String||Technical name of an indicator||Make sure to follow the [[Coding_rules|naming convention]] for technical names in WeSIS and WeSISpedia.
|-
|value||5||yes||Numeric, string or date||The actual value of the indicator||Always use the dot (.) as the decimal separator for numeric data. For dates follow the [[Coding_rules|date time format]].
|-
|user_na_values||6||yes||Symbol||Flag for "missing data"||Use an exclamation mark (!) to flag any value(s) as [[File_upload_guide#Uploading_.22missing.22_or_N.2FA_values|"missing data"]]. The default is a blank cell/column.
|-
|unit||7||yes||String||The unit of ''value''||For dates the unit is "date", otherwise name the actual unit of the value (e.g. "per 1000 inhabitants", "as % of GDP" etc.)
|-
|scale||8||yes||String||The scale of ''value''||Only one of six possible values is allowed here:
* Binary1
* Multinomial
* Ordinal
* Metric
* String
* Date
|-
|source||9||partly||String||The source of ''value''||If ''value'' derives from a common source please check if there is already a [[Abbreviations|harmonized abbreviation]].
|-
|publication_date||10||yes||Date||Date of data collection and/or upload to WeSIS||This date refers to the data collection, the last check or the upload, i.e. it does ''not'' refer to ''year'' or date of ''value''. Yet, the common [[Coding_rules|date time format]] applies accordingly.
|-
|category||11||partly||String||Name of the category an indicator belongs to||Category refers to the upper level an indicator belongs to ("parent topic", i.e., one of either a policy field, domestic condition or relation).
|-
|label||12||no||String||Label of the (technical) category name||This is an easy-to-read label of the indicator. It is the same as "indicator name" in WeSISpedia and ''label'' in the Quick info box of an indicator page in WeSISpedia.
|-
|plausibility_concerns||13||yes||String||[[Data quality]] aspect||Responses to the question "Are there notable concerns about the plausibility of this data point?"; only one of six answers is allowed here:
*No
*Yes, from a time-series perspective
*Yes, from a cross-sectional perspective
*Yes, due to limited sources
*Yes, due to contradicting sources
*Other
|}

1 Note that binary is used as a strict boolean type with allowed values of <code>0/1</code> or <code>true/false</code>; if you have a binary indicator but want to use "user-defined missing values", you have to choose "Multinomial" as scale.

== Mandatory columns for the dyadic data template ==

{|class="wikitable sortable" border="1"
!Column name!!Column number!!Standardized?!!Type||Description!!Comments
|-
|cow_code_sender||1||yes||Numeric||Country code of the "sender" according to the COW scheme||"Sender" is used here synonymously to any node in undirected networks. See the list of [[Country_codes|country and IO codes]] used for WeSIS and WeSISpedia.
|-
|country_name_sender||2||yes||String||Country name of the "sender"||See the list of [[Country_codes|country and IO codes]] used for WeSIS and WeSISpedia.
|-
|cow_code_receiver||3||yes||Numeric||Country code of the "receiver" according to the COW scheme||"Receiver" is used here synonymously to any node in undirected networks. See the list of [[Country_codes|country and IO codes]] used for WeSIS and WeSISpedia.
|-
|country_name_receiver||4||yes||String||Country name of the "receiver"||See the list of [[Country_codes|country and IO codes]] used for WeSIS and WeSISpedia.
|-
|year||5||yes||Numeric||The year ''value'' refers to||Note that a date, like election dates or the date of introduction of a policy, is a ''value''. ''year'', thus, may "double" the value.
|-
|technical_variable_name||6||partly||String||Technical name of an indicator||Make sure to follow the [[Coding_rules|naming convention]] for technical names in WeSIS and WeSISpedia.
|-
|value||7||yes||Numeric, string or date time||The actual value of the indicator||Always use the dot (.) as the decimal separator for numeric data. For dates follow the [[Coding_rules|date time format]].
|-
|user_na_values||8||yes||Symbol||Flag for "missing data"||Use an exclamation mark (!) to flag any value(s) as [[File_upload_guide#Uploading_.22missing.22_or_N.2FA_values|"missing data"]]. The default is a blank cell/column.
|-
|unit||9||yes||String||The unit of ''value''||For dates the unit is "date", otherwise name the actual unit of the value (e.g. "per 1000 inhabitants", "as % of GDP" etc.)
|-
|scale||10||yes||String||The scale of ''value''||Only one of six possible values is allowed here:
* Binary1
* Multinomial
* Ordinal
* Metric
* String
* Date
|-
|source||11||partly||String||The source of ''value''||If ''value'' derives from a common source please check if there is already a [[Abbreviations|harmonized abbreviation]].
|-
|publication_date||12||yes||Date||Date of data collection and/or upload to WeSIS||This date refers to the data collection, the last check or the upload, i.e. it does ''not'' refer to ''year'' or date of ''value''. Yet, the common [[Coding_rules|date time format]] applies accordingly.
|-
|category||13||partly||String||Name of the category an indicator belongs to||Category refers to the upper level an indicator belongs to ("parent topic", i.e., one of either a policy field, domestic condition or relation).
|-
|label||14||no||String||Label of the (technical) category name||This is an easy-to-read label of the indicator. It is the same as "indicator name" in WeSISpedia and ''label'' in the Quick info box of an indicator page in WeSISpedia.
|-
|plausibility_concerns||15||yes||String||[[Data quality]] aspect||Responses to the question "Are there notable concerns about the plausibility of this data point?"; only one of six answers is allowed here:
*No
*Yes, from a time-series perspective
*Yes, from a cross-sectional perspective
*Yes, due to limited sources
*Yes, due to contradicting sources
*Other
|}

1 Note that binary is used as a strict boolean type with allowed values of <code>0/1</code> or <code>true/false</code>; if you have a binary indicator but want to use "user-defined missing values", you have to choose "Multinomial" as scale.

== Misc ==

Contributors: Nils Düpont, Sebastian Haunss, Gabriela León, Gabriella Skitalinska, Nate Breznau, Mohamad Tofayel Ahmed

Revisions:
* CRC internal release of templates on March 06, 2019
* Update of the template on November 06, 2023

Social structure

2024-06-06T11:53:29Z

Hsalzmann: