diff --git a/statvar_imports/eurostat/live_births_total_by_month/README.md b/statvar_imports/eurostat/live_births_total_by_month/README.md new file mode 100644 index 0000000000..e8da14e735 --- /dev/null +++ b/statvar_imports/eurostat/live_births_total_by_month/README.md @@ -0,0 +1,87 @@ +# Eurostat Live Births Total By Month Import + +## Overview +This dataset contains monthly live births data at national and regional levels, sourced from Eurostat. The data tracks the total number of live births per month across various European countries and regions. + +**type of place:** Country, NUTS Regions (Level 0-3) +**years:** Historical data to present (1960-2025) +**place_resolution:** Resolved to DCIDs (e.g., dcid:country/FRA, dcid:nuts/AT113) + +## Data Source +**Source URL:** +https://ec.europa.eu/eurostat/databrowser/view/DEMO_FMONTH__custom_270818/default/table?lang=en + +**Provenance Description:** +The data is provided by Eurostat, the statistical office of the European Union. It is part of the "Demography and migration" database, specifically the "Live births by month" (DEMO_FMONTH) dataset. + +## Refresh Type +Automatic Refresh + +The refresh is automated using the provided `run.sh` script, which handles both data download and processing. + +## How To Run Import +To execute the complete import process (download and processing), run: +```bash +./run.sh +``` + +### Script Details: +- **Download**: Uses `curl` to fetch the latest SDMX-CSV data from Eurostat's dissemination API. +- **Processing**: Uses `stat_var_processor.py` to map raw data to Data Commons StatVarObservations using the PV map and metadata configuration. + +## Processing Instructions +To process the Eurostat Live Births data and generate statistical variables, use the following commands from your current import data directory: + +Download input file + +```bash +mkdir -p input_files +curl -L --retry 3 "https://ec.europa.eu/eurostat/api/dissemination/sdmx/2.1/data/DEMO_FMONTH/?format=SDMX-CSV&compressed=false" -o input_files/live_births_total_by_month_data_input.csv +``` + +For Test Data Run + +```bash +python3 ../../../tools/statvar_importer/stat_var_processor.py \ + "--input_data=./test_data/live_births_total_by_month_data_input.csv" \ + "--pv_map=./live_births_total_by_month_pvmap.csv" \ + "--output_path=./test_data/live_births_total_by_month_output" \ + "--config_file=./live_births_total_by_month_metadata.csv" \ + "--existing_statvar_mcf=gs://unresolved_mcf/scripts/statvar/stat_vars.mcf" +``` + +For Main data processing run + +```bash +python3 ../../../tools/statvar_importer/stat_var_processor.py \ + "--input_data=./input_files/*.csv" \ + "--pv_map=./live_births_total_by_month_pvmap.csv" \ + "--config_file=./live_births_total_by_month_metadata.csv" \ + "--generate_statvar_name=True" \ + "--skip_constant_csv_columns=False" \ + "--output_columns=observationDate,observationAbout,variableMeasured,value,observationPeriod,unit" \ + "--output_path=./live_births_total_by_month_output" \ + "--places_resolved_csv=./places_resolved.csv" \ + "--existing_statvar_mcf=gs://unresolved_mcf/scripts/statvar/stat_vars.mcf" +``` + +## Key Files +- `run.sh`: Main execution script for download and processing. +- `live_births_total_by_month_pvmap.csv`: Property-Value mapping for StatVar definitions and dimensions. +- `live_births_total_by_month_metadata.csv`: Configuration parameters for the processor. +- `places_resolved.csv`: Mapping of place codes to Data Commons DCIDs. +- `live_births_total_by_month_output.csv`: Processed statistical observations. +- `live_births_total_by_month_output.tmcf`: Template MCF mapping the CSV columns to Data Commons schema. + +## Validation +To validate the generated data, use the Data Commons import tool (lint mode): +```bash +java -jar datacommons-import-tool.jar lint live_births_total_by_month_output.csv live_births_total_by_month_output.tmcf +``` +The resulting reports (`report.json`, `summary_report.html`) in `dc_generated/` provide detailed insights into data quality and validation status. + +## Testing +Testing is performed using the `test_data` directory: +- Raw Input: `test_data/live_births_total_by_month_data_input.csv` +- Expected Output: `test_data/live_births_total_by_month_output.csv` +- Expected TMCF: `test_data/live_births_total_by_month_output.tmcf` diff --git a/statvar_imports/eurostat/live_births_total_by_month/live_births_total_by_month_metadata.csv b/statvar_imports/eurostat/live_births_total_by_month/live_births_total_by_month_metadata.csv new file mode 100644 index 0000000000..cb44b3b4d3 --- /dev/null +++ b/statvar_imports/eurostat/live_births_total_by_month/live_births_total_by_month_metadata.csv @@ -0,0 +1,4 @@ +parameter,value +header_rows,1 +#input_rows,20 + diff --git a/statvar_imports/eurostat/live_births_total_by_month/live_births_total_by_month_pvmap.csv b/statvar_imports/eurostat/live_births_total_by_month/live_births_total_by_month_pvmap.csv new file mode 100644 index 0000000000..1463b616bc --- /dev/null +++ b/statvar_imports/eurostat/live_births_total_by_month/live_births_total_by_month_pvmap.csv @@ -0,0 +1,49 @@ +key,property1,value1,property2,value2,property3,value3,property4,value4 + +# Global Dataflow properties (Standard Data Commons Demographic Modeling) +"DATAFLOW:ESTAT:DEMO_FMONTH(1.0)",populationType,dcs:BirthEvent,measuredProperty,dcs:count,statType,dcs:measuredValue,, + +# --- AUTOMATED CLEANUP: Ignored Columns --- +LAST UPDATE,#ignore,"",,,,,, +freq,#ignore,"",,,,,, +CONF_STATUS,#ignore,"",,,,,, + +# Month mapping & observationPeriod variable passing (Grounded on standard ISO-8601 durations) +month:M01,Month,01,observationPeriod,P1M,,,, +month:M02,Month,02,observationPeriod,P1M,,,, +month:M03,Month,03,observationPeriod,P1M,,,, +month:M04,Month,04,observationPeriod,P1M,,,, +month:M05,Month,05,observationPeriod,P1M,,,, +month:M06,Month,06,observationPeriod,P1M,,,, +month:M07,Month,07,observationPeriod,P1M,,,, +month:M08,Month,08,observationPeriod,P1M,,,, +month:M09,Month,09,observationPeriod,P1M,,,, +month:M10,Month,10,observationPeriod,P1M,,,, +month:M11,Month,11,observationPeriod,P1M,,,, +month:M12,Month,12,observationPeriod,P1M,,,, +month:TOTAL,Month,"",observationPeriod,P1Y,,,, +month:UNK,#ignore,"",,,,,, + +# Unit dimension (dcs:Person is standard for demographic counts) +unit:NR,unit,dcs:Person,,,,,, + +# Time column capture (Stores year for the strict date checker) +TIME_PERIOD,Year,{Data},,,,,, + +geo,observationAbout,{Data},,,,,, + +# Measure and Date Resolution +OBS_VALUE,#Eval,"observationDate='{Year}-{Month}' if '{Month}' else '{Year}'",value,{Number},,,, + +# Safely Ignoring Transient and Composite Status Flags to Prevent Ingestion Warnings +OBS_FLAG:b,#ignore,"",,,,,, +OBS_FLAG:e,#ignore,"",,,,,, +OBS_FLAG:p,#ignore,"",,,,,, +OBS_FLAG:f,#ignore,"",,,,,, +OBS_FLAG:n,#ignore,"",,,,,, +OBS_FLAG:u,#ignore,"",,,,,, +OBS_FLAG:be,#ignore,"",,,,,, +OBS_FLAG:bep,#ignore,"",,,,,, + +# Missing values +OBS_FLAG:M,MissingValue,"",,,,,, diff --git a/statvar_imports/eurostat/live_births_total_by_month/manifest.json b/statvar_imports/eurostat/live_births_total_by_month/manifest.json new file mode 100644 index 0000000000..193ac5b8f3 --- /dev/null +++ b/statvar_imports/eurostat/live_births_total_by_month/manifest.json @@ -0,0 +1,34 @@ +{ + "import_specifications": [ + { + "import_name": "EuroStat_Live_Births_Total_By_Month", + "curator_emails": [ + "support@datacommons.org" + ], + "provenance_url": "https://ec.europa.eu/eurostat/databrowser/view/DEMO_FMONTH__custom_270818/default/table?lang=en", + "provenance_description": "Annual tracking of the number of live births by month across reporting European countries", + "scripts": [ + "run.sh" + ], + "import_inputs": [ + { + "template_mcf": "live_births_total_by_month_output.tmcf", + "cleaned_csv": "live_births_total_by_month_output.csv" + } + ], + "source_files": [ + "input_files/*.csv" + ], + "cron_schedule": "5 1 1,15 * *", + "resource_limits": {"cpu": 4, "memory": 8, "disk":100}, + "config_override": { + "invoke_import_validation": true, + "invoke_import_tool": true, + "invoke_differ_tool": true, + "skip_input_upload": false, + "skip_gcs_upload": false, + "cleanup_gcs_volume_mount": false + } + } + ] +} diff --git a/statvar_imports/eurostat/live_births_total_by_month/places_resolved.csv b/statvar_imports/eurostat/live_births_total_by_month/places_resolved.csv new file mode 100644 index 0000000000..acc65e505e --- /dev/null +++ b/statvar_imports/eurostat/live_births_total_by_month/places_resolved.csv @@ -0,0 +1,265 @@ +"place_name","dcid","placeId","wikidataId","name","alternateName","typeOf","containedInPlace" +"AD","dcid:country/AND","","","","","","" +"AE","dcid:country/ARE","","","","","","" +"AF","dcid:country/AFG","","","","","","" +"AG","dcid:country/ATG","","","","","","" +"AI","dcid:country/AIA","","","","","","" +"AL","dcid:country/ALB","","","","","","" +"AM","dcid:country/ARM","","","","","","" +"AN","dcid:country/ANT","","","","","","" +"AO","dcid:country/AGO","","","","","","" +"AQ","dcid:country/ATA","","","","","","" +"AR","dcid:country/ARG","","","","","","" +"AS","dcid:country/ASM","","","","","","" +"AT","dcid:country/AUT","","","","","","" +"AU","dcid:country/AUS","","","","","","" +"AW","dcid:country/ABW","","","","","","" +"AZ","dcid:country/AZE","","","","","","" +"BA","dcid:country/BIH","","","","","","" +"BB","dcid:country/BRB","","","","","","" +"BD","dcid:country/BGD","","","","","","" +"BE","dcid:country/BEL","","","","","","" +"BF","dcid:country/BFA","","","","","","" +"BG","dcid:country/BGR","","","","","","" +"BH","dcid:country/BHR","","","","","","" +"BI","dcid:country/BDI","","","","","","" +"BJ","dcid:country/BEN","","","","","","" +"BL","dcid:country/BLM","","","","","","" +"BM","dcid:country/BMU","","","","","","" +"BN","dcid:country/BRN","","","","","","" +"BO","dcid:country/BOL","","","","","","" +"BR","dcid:country/BRA","","","","","","" +"BS","dcid:country/BHS","","","","","","" +"BT","dcid:country/BTN","","","","","","" +"BV","dcid:country/BVT","","","","","","" +"BW","dcid:country/BWA","","","","","","" +"BY","dcid:country/BLR","","","","","","" +"BZ","dcid:country/BLZ","","","","","","" +"CA","dcid:country/CAN","","","","","","" +"CC","dcid:country/CCK","","","","","","" +"CD","dcid:country/COD","","","","","","" +"CF","dcid:country/CAF","","","","","","" +"CG","dcid:country/COG","","","","","","" +"CH","dcid:country/CHE","","","","","","" +"CI","dcid:country/CIV","","","","","","" +"CK","dcid:country/COK","","","","","","" +"CL","dcid:country/CHL","","","","","","" +"CM","dcid:country/CMR","","","","","","" +"CN","dcid:country/CHN","","","","","","" +"CO","dcid:country/COL","","","","","","" +"CR","dcid:country/CRI","","","","","","" +"CU","dcid:country/CUB","","","","","","" +"CV","dcid:country/CPV","","","","","","" +"CX","dcid:country/CXR","","","","","","" +"CY","dcid:country/CYP","","","","","","" +"CZ","dcid:country/CZE","","","","","","" +"DE","dcid:country/DEU","","","","","","" +"DJ","dcid:country/DJI","","","","","","" +"DK","dcid:country/DNK","","","","","","" +"DM","dcid:country/DMA","","","","","","" +"DO","dcid:country/DOM","","","","","","" +"DZ","dcid:country/DZA","","","","","","" +"EC","dcid:country/ECU","","","","","","" +"EE","dcid:country/EST","","","","","","" +"EG","dcid:country/EGY","","","","","","" +"EH","dcid:country/ESH","","","","","","" +"EL","dcid:country/GRC","","","","","","" +"ER","dcid:country/ERI","","","","","","" +"ES","dcid:country/ESP","","","","","","" +"ET","dcid:country/ETH","","","","","","" +"FI","dcid:country/FIN","","","","","","" +"FJ","dcid:country/FJI","","","","","","" +"FK","dcid:country/FLK","","","","","","" +"FM","dcid:country/FSM","","","","","","" +"FO","dcid:country/FRO","","","","","","" +"FR","dcid:country/FRA","","","","","","" +"FX","dcid:country/FXX","","","","","","" +"GA","dcid:country/GAB","","","","","","" +"UK","dcid:country/GBR","","","","","","" +"GD","dcid:country/GRD","","","","","","" +"GE","dcid:country/GEO","","","","","","" +"GF","dcid:country/GUF","","","","","","" +"GG","dcid:country/GGY","","","","","","" +"GH","dcid:country/GHA","","","","","","" +"GI","dcid:country/GIB","","","","","","" +"GL","dcid:country/GRL","","","","","","" +"GM","dcid:country/GMB","","","","","","" +"GN","dcid:country/GIN","","","","","","" +"GQ","dcid:country/GNQ","","","","","","" +"GS","dcid:country/SGS","","","","","","" +"GT","dcid:country/GTM","","","","","","" +"GU","dcid:country/GUM","","","","","","" +"GW","dcid:country/GNB","","","","","","" +"GY","dcid:country/GUY","","","","","","" +"HK","dcid:country/HKG","","","","","","" +"HM","dcid:country/HMD","","","","","","" +"HN","dcid:country/HND","","","","","","" +"HR","dcid:country/HRV","","","","","","" +"HT","dcid:country/HTI","","","","","","" +"HU","dcid:country/HUN","","","","","","" +"ID","dcid:country/IDN","","","","","","" +"IE","dcid:country/IRL","","","","","","" +"IL","dcid:country/ISR","","","","","","" +"IM","dcid:country/IMN","","","","","","" +"IN","dcid:country/IND","","","","","","" +"IO","dcid:country/IOT","","","","","","" +"IQ","dcid:country/IRQ","","","","","","" +"IR","dcid:country/IRN","","","","","","" +"IS","dcid:country/ISL","","","","","","" +"IT","dcid:country/ITA","","","","","","" +"JE","dcid:country/JEY","","","","","","" +"JM","dcid:country/JAM","","","","","","" +"JO","dcid:country/JOR","","","","","","" +"JP","dcid:country/JPN","","","","","","" +"KE","dcid:country/KEN","","","","","","" +"KG","dcid:country/KGZ","","","","","","" +"KH","dcid:country/KHM","","","","","","" +"KI","dcid:country/KIR","","","","","","" +"KM","dcid:country/COM","","","","","","" +"KN","dcid:country/KNA","","","","","","" +"KP","dcid:country/PRK","","","","","","" +"KR","dcid:country/KOR","","","","","","" +"KW","dcid:country/KWT","","","","","","" +"KY","dcid:country/CYM","","","","","","" +"KZ","dcid:country/KAZ","","","","","","" +"LA","dcid:country/LAO","","","","","","" +"LB","dcid:country/LBN","","","","","","" +"LC","dcid:country/LCA","","","","","","" +"LI","dcid:country/LIE","","","","","","" +"LK","dcid:country/LKA","","","","","","" +"LR","dcid:country/LBR","","","","","","" +"LS","dcid:country/LSO","","","","","","" +"LT","dcid:country/LTU","","","","","","" +"LU","dcid:country/LUX","","","","","","" +"LV","dcid:country/LVA","","","","","","" +"LY","dcid:country/LBY","","","","","","" +"MA","dcid:country/MAR","","","","","","" +"MC","dcid:country/MCO","","","","","","" +"MD","dcid:country/MDA","","","","","","" +"ME","dcid:country/MNE","","","","","","" +"MF","dcid:country/MAF","","","","","","" +"MG","dcid:country/MDG","","","","","","" +"MH","dcid:country/MHL","","","","","","" +"MK","dcid:country/MKD","","","","","","" +"ML","dcid:country/MLI","","","","","","" +"MM","dcid:country/MMR","","","","","","" +"MN","dcid:country/MNG","","","","","","" +"MO","dcid:country/MAC","","","","","","" +"MP","dcid:country/MNP","","","","","","" +"MQ","dcid:country/MTQ","","","","","","" +"MR","dcid:country/MRT","","","","","","" +"MS","dcid:country/MSR","","","","","","" +"MT","dcid:country/MLT","","","","","","" +"MU","dcid:country/MUS","","","","","","" +"MV","dcid:country/MDV","","","","","","" +"MW","dcid:country/MWI","","","","","","" +"MX","dcid:country/MEX","","","","","","" +"MY","dcid:country/MYS","","","","","","" +"MZ","dcid:country/MOZ","","","","","","" +"NA","dcid:country/NAM","","","","","","" +"NC","dcid:country/NCL","","","","","","" +"NE","dcid:country/NER","","","","","","" +"NF","dcid:country/NFK","","","","","","" +"NG","dcid:country/NGA","","","","","","" +"NI","dcid:country/NIC","","","","","","" +"NL","dcid:country/NLD","","","","","","" +"NO","dcid:country/NOR","","","","","","" +"NP","dcid:country/NPL","","","","","","" +"NR","dcid:country/NRU","","","","","","" +"NU","dcid:country/NIU","","","","","","" +"NZ","dcid:country/NZL","","","","","","" +"OM","dcid:country/OMN","","","","","","" +"PA","dcid:country/PAN","","","","","","" +"PE","dcid:country/PER","","","","","","" +"PF","dcid:country/PYF","","","","","","" +"PG","dcid:country/PNG","","","","","","" +"PH","dcid:country/PHL","","","","","","" +"PK","dcid:country/PAK","","","","","","" +"PL","dcid:country/POL","","","","","","" +"PM","dcid:country/SPM","","","","","","" +"PN","dcid:country/PCN","","","","","","" +"PR","dcid:country/PRI","","","","","","" +"PS","dcid:country/PSE","","","","","","" +"PT","dcid:country/PRT","","","","","","" +"PW","dcid:country/PLW","","","","","","" +"PY","dcid:country/PRY","","","","","","" +"QA","dcid:country/QAT","","","","","","" +"RE","dcid:country/REU","","","","","","" +"RO","dcid:country/ROU","","","","","","" +"RS","dcid:country/SRB","","","","","","" +"RU","dcid:country/RUS","","","","","","" +"RW","dcid:country/RWA","","","","","","" +"SA","dcid:country/SAU","","","","","","" +"SB","dcid:country/SLB","","","","","","" +"SC","dcid:country/SYC","","","","","","" +"SD","dcid:country/SDN","","","","","","" +"SE","dcid:country/SWE","","","","","","" +"SG","dcid:country/SGP","","","","","","" +"SH","dcid:country/SHN","","","","","","" +"SI","dcid:country/SVN","","","","","","" +"SJ","dcid:country/SJM","","","","","","" +"SK","dcid:country/SVK","","","","","","" +"SL","dcid:country/SLE","","","","","","" +"SM","dcid:country/SMR","","","","","","" +"SN","dcid:country/SEN","","","","","","" +"SO","dcid:country/SOM","","","","","","" +"SR","dcid:country/SUR","","","","","","" +"ST","dcid:country/STP","","","","","","" +"SV","dcid:country/SLV","","","","","","" +"SX","dcid:country/SXM","","","","","","" +"SY","dcid:country/SYR","","","","","","" +"SZ","dcid:country/SWZ","","","","","","" +"TC","dcid:country/TCA","","","","","","" +"TD","dcid:country/TCD","","","","","","" +"TF","dcid:country/ATF","","","","","","" +"TG","dcid:country/TGO","","","","","","" +"TH","dcid:country/THA","","","","","","" +"TJ","dcid:country/TJK","","","","","","" +"TK","dcid:country/TKL","","","","","","" +"TL","dcid:country/TLS","","","","","","" +"TM","dcid:country/TKM","","","","","","" +"TN","dcid:country/TUN","","","","","","" +"TO","dcid:country/TON","","","","","","" +"TR","dcid:country/TUR","","","","","","" +"TT","dcid:country/TTO","","","","","","" +"TV","dcid:country/TUV","","","","","","" +"TW","dcid:country/TWN","","","","","","" +"TZ","dcid:country/TZA","","","","","","" +"UA","dcid:country/UKR","","","","","","" +"UG","dcid:country/UGA","","","","","","" +"UM","dcid:country/UMI","","","","","","" +"US","dcid:country/USA","","","","","","" +"UY","dcid:country/URY","","","","","","" +"UZ","dcid:country/UZB","","","","","","" +"VA","dcid:country/VAT","","","","","","" +"VC","dcid:country/VCT","","","","","","" +"VE","dcid:country/VEN","","","","","","" +"VG","dcid:country/VGB","","","","","","" +"VI","dcid:country/VIR","","","","","","" +"VN","dcid:country/VNM","","","","","","" +"VU","dcid:country/VUT","","","","","","" +"WF","dcid:country/WLF","","","","","","" +"WS","dcid:country/WSM","","","","","","" +"XK","dcid:country/XKS","","","","","","" +"YE","dcid:country/YEM","","","","","","" +"YT","dcid:country/MYT","","","","","","" +"ZA","dcid:country/ZAF","","","","","","" +"ZM","dcid:country/ZMB","","","","","","" +"ZW","dcid:country/ZWE","","","","","","" +"DE_TOT","dcid:nuts/DE_TOT","","","","","","" +"EA","dcid:nuts/EA","","","","","","" +"EA12","dcid:nuts/EA12","","","","","","" +"EA18","dcid:nuts/EA18","","","","","","" +"EA19","dcid:nuts/EA19","","","","","","" +"EEA","dcid:nuts/EEA","","","","","","" +"EEA30_2007","dcid:nuts/EEA30_2007","","","","","","" +"EEA31","dcid:nuts/EEA31","","","","","","" +"EFTA","dcid:nuts/EFTA","","","","","","" +"EU","dcid:nuts/EU","","","","","","" +"EU27","dcid:nuts/EU27","","","","","","" +"EU27_2007","dcid:nuts/EU27_2007","","","","","","" +"EU27_2020","dcid:nuts/EU27_2020","","","","","","" +"EA20","dcid:nuts/EA20","","","","","","" +"EA21","dcid:nuts/EA21","","","","","","" +"EU28","dcid:nuts/EU28","","","","","","" +"EEA28","dcid:nuts/EEA28","","","","","","" \ No newline at end of file diff --git a/statvar_imports/eurostat/live_births_total_by_month/run.sh b/statvar_imports/eurostat/live_births_total_by_month/run.sh new file mode 100644 index 0000000000..5d4d1dde15 --- /dev/null +++ b/statvar_imports/eurostat/live_births_total_by_month/run.sh @@ -0,0 +1,19 @@ +#!/bin/bash + +# Step 1: Data Download +mkdir -p input_files +curl -L --retry 3 "https://ec.europa.eu/eurostat/api/dissemination/sdmx/2.1/data/DEMO_FMONTH/?format=SDMX-CSV&compressed=false" -o ./input_files/live_births_total_by_month_data_input.csv + +# Step 2: Data Processing + +python3 ../../../tools/statvar_importer/stat_var_processor.py \ + "--input_data=./input_files/*.csv" \ + "--pv_map=./live_births_total_by_month_pvmap.csv" \ + "--config_file=./live_births_total_by_month_metadata.csv" \ + "--generate_statvar_name=True" \ + "--skip_constant_csv_columns=False" \ + "--output_columns=observationDate,observationAbout,variableMeasured,value,observationPeriod,unit" \ + "--output_path=./live_births_total_by_month_output" \ + "--places_resolved_csv=./places_resolved.csv" \ + "--existing_statvar_mcf=gs://unresolved_mcf/scripts/statvar/stat_vars.mcf" + \ No newline at end of file diff --git a/statvar_imports/eurostat/live_births_total_by_month/test_data/live_births_total_by_month_data_input.csv b/statvar_imports/eurostat/live_births_total_by_month/test_data/live_births_total_by_month_data_input.csv new file mode 100644 index 0000000000..e348119eea --- /dev/null +++ b/statvar_imports/eurostat/live_births_total_by_month/test_data/live_births_total_by_month_data_input.csv @@ -0,0 +1,100 @@ +DATAFLOW,LAST UPDATE,freq,unit,month,geo,TIME_PERIOD,OBS_VALUE,OBS_FLAG,CONF_STATUS +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AD,2005,69,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AD,2006,52,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AD,2007,78,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AD,2008,67,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AD,2009,69,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AD,2010,69,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AD,2011,68,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AD,2012,59,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AD,2018,40,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AL,1997,4743,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AL,1998,4743,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AL,2004,3588,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AL,2005,3092,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AL,2006,3191,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AL,2007,2893,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AL,2013,2839,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AL,2014,2837,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AL,2015,2742,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AL,2016,2517,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AL,2017,2423,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AL,2018,2468,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AL,2019,2393,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AL,2020,2316,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AL,2021,2066,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AL,2022,2277,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AL,2023,1991,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AL,2024,1985,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AM,2005,3319,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AM,2006,3354,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AM,2007,3362,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AM,2008,3389,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AM,2009,3715,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AM,2014,3597,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AM,2015,3444,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AM,2016,4009,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AM,2017,3081,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AM,2018,3031,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AM,2019,3020,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AM,2020,2751,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AM,2022,2890,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AM,2023,3275,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AT,1960,10666,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AT,1961,11372,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AT,1962,11433,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AT,1963,11805,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AT,1964,11343,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AT,1965,11378,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AT,1966,10723,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AT,1967,10993,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AT,1968,11197,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AT,1969,10987,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AT,1970,9759,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AT,1971,9507,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AT,1972,9187,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AT,1973,8750,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AT,1974,8545,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AT,1975,8343,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AT,1976,8047,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AT,1977,7259,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AT,1978,7157,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AT,1979,7554,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AT,1980,7481,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AT,1981,7890,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AT,1982,8142,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AT,1983,7825,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AT,1984,7424,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AT,1985,7546,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AT,1986,7468,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AT,1987,7279,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AT,1988,7442,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AT,1989,7470,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AT,1990,7612,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AT,1991,7925,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AT,1992,7932,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AT,1993,8266,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AT,1994,7798,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AT,1995,7603,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AT,1996,7436,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AT,1997,7318,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AT,1998,6860,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AT,1999,6697,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AT,2000,6587,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AT,2001,6611,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AT,2002,6594,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AT,2003,6570,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AT,2004,6780,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AT,2005,6482,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AT,2006,6526,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AT,2007,6504,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AT,2008,6739,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AT,2009,6475,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AT,2010,6414,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AT,2011,6651,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AT,2012,6389,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AT,2013,6629,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AT,2014,6666,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AT,2015,6985,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AT,2016,6980,, +ESTAT:DEMO_FMONTH(1.0),20/04/26 23:00:00,A,NR,M01,AT,2017,7040,, diff --git a/statvar_imports/eurostat/live_births_total_by_month/test_data/live_births_total_by_month_output.csv b/statvar_imports/eurostat/live_births_total_by_month/test_data/live_births_total_by_month_output.csv new file mode 100644 index 0000000000..aaec54d7a4 --- /dev/null +++ b/statvar_imports/eurostat/live_births_total_by_month/test_data/live_births_total_by_month_output.csv @@ -0,0 +1,100 @@ +observationDate,observationAbout,variableMeasured,value,observationPeriod,unit +2005-01,dcid:country/AND,dcid:Count_BirthEvent,69,P1M,dcs:Person +2006-01,dcid:country/AND,dcid:Count_BirthEvent,52,P1M,dcs:Person +2007-01,dcid:country/AND,dcid:Count_BirthEvent,78,P1M,dcs:Person +2008-01,dcid:country/AND,dcid:Count_BirthEvent,67,P1M,dcs:Person +2009-01,dcid:country/AND,dcid:Count_BirthEvent,69,P1M,dcs:Person +2010-01,dcid:country/AND,dcid:Count_BirthEvent,69,P1M,dcs:Person +2011-01,dcid:country/AND,dcid:Count_BirthEvent,68,P1M,dcs:Person +2012-01,dcid:country/AND,dcid:Count_BirthEvent,59,P1M,dcs:Person +2018-01,dcid:country/AND,dcid:Count_BirthEvent,40,P1M,dcs:Person +1997-01,dcid:country/ALB,dcid:Count_BirthEvent,4743,P1M,dcs:Person +1998-01,dcid:country/ALB,dcid:Count_BirthEvent,4743,P1M,dcs:Person +2004-01,dcid:country/ALB,dcid:Count_BirthEvent,3588,P1M,dcs:Person +2005-01,dcid:country/ALB,dcid:Count_BirthEvent,3092,P1M,dcs:Person +2006-01,dcid:country/ALB,dcid:Count_BirthEvent,3191,P1M,dcs:Person +2007-01,dcid:country/ALB,dcid:Count_BirthEvent,2893,P1M,dcs:Person +2013-01,dcid:country/ALB,dcid:Count_BirthEvent,2839,P1M,dcs:Person +2014-01,dcid:country/ALB,dcid:Count_BirthEvent,2837,P1M,dcs:Person +2015-01,dcid:country/ALB,dcid:Count_BirthEvent,2742,P1M,dcs:Person +2016-01,dcid:country/ALB,dcid:Count_BirthEvent,2517,P1M,dcs:Person +2017-01,dcid:country/ALB,dcid:Count_BirthEvent,2423,P1M,dcs:Person +2018-01,dcid:country/ALB,dcid:Count_BirthEvent,2468,P1M,dcs:Person +2019-01,dcid:country/ALB,dcid:Count_BirthEvent,2393,P1M,dcs:Person +2020-01,dcid:country/ALB,dcid:Count_BirthEvent,2316,P1M,dcs:Person +2021-01,dcid:country/ALB,dcid:Count_BirthEvent,2066,P1M,dcs:Person +2022-01,dcid:country/ALB,dcid:Count_BirthEvent,2277,P1M,dcs:Person +2023-01,dcid:country/ALB,dcid:Count_BirthEvent,1991,P1M,dcs:Person +2024-01,dcid:country/ALB,dcid:Count_BirthEvent,1985,P1M,dcs:Person +2005-01,dcid:country/ARM,dcid:Count_BirthEvent,3319,P1M,dcs:Person +2006-01,dcid:country/ARM,dcid:Count_BirthEvent,3354,P1M,dcs:Person +2007-01,dcid:country/ARM,dcid:Count_BirthEvent,3362,P1M,dcs:Person +2008-01,dcid:country/ARM,dcid:Count_BirthEvent,3389,P1M,dcs:Person +2009-01,dcid:country/ARM,dcid:Count_BirthEvent,3715,P1M,dcs:Person +2014-01,dcid:country/ARM,dcid:Count_BirthEvent,3597,P1M,dcs:Person +2015-01,dcid:country/ARM,dcid:Count_BirthEvent,3444,P1M,dcs:Person +2016-01,dcid:country/ARM,dcid:Count_BirthEvent,4009,P1M,dcs:Person +2017-01,dcid:country/ARM,dcid:Count_BirthEvent,3081,P1M,dcs:Person +2018-01,dcid:country/ARM,dcid:Count_BirthEvent,3031,P1M,dcs:Person +2019-01,dcid:country/ARM,dcid:Count_BirthEvent,3020,P1M,dcs:Person +2020-01,dcid:country/ARM,dcid:Count_BirthEvent,2751,P1M,dcs:Person +2022-01,dcid:country/ARM,dcid:Count_BirthEvent,2890,P1M,dcs:Person +2023-01,dcid:country/ARM,dcid:Count_BirthEvent,3275,P1M,dcs:Person +1960-01,dcid:country/AUT,dcid:Count_BirthEvent,10666,P1M,dcs:Person +1961-01,dcid:country/AUT,dcid:Count_BirthEvent,11372,P1M,dcs:Person +1962-01,dcid:country/AUT,dcid:Count_BirthEvent,11433,P1M,dcs:Person +1963-01,dcid:country/AUT,dcid:Count_BirthEvent,11805,P1M,dcs:Person +1964-01,dcid:country/AUT,dcid:Count_BirthEvent,11343,P1M,dcs:Person +1965-01,dcid:country/AUT,dcid:Count_BirthEvent,11378,P1M,dcs:Person +1966-01,dcid:country/AUT,dcid:Count_BirthEvent,10723,P1M,dcs:Person +1967-01,dcid:country/AUT,dcid:Count_BirthEvent,10993,P1M,dcs:Person +1968-01,dcid:country/AUT,dcid:Count_BirthEvent,11197,P1M,dcs:Person +1969-01,dcid:country/AUT,dcid:Count_BirthEvent,10987,P1M,dcs:Person +1970-01,dcid:country/AUT,dcid:Count_BirthEvent,9759,P1M,dcs:Person +1971-01,dcid:country/AUT,dcid:Count_BirthEvent,9507,P1M,dcs:Person +1972-01,dcid:country/AUT,dcid:Count_BirthEvent,9187,P1M,dcs:Person +1973-01,dcid:country/AUT,dcid:Count_BirthEvent,8750,P1M,dcs:Person +1974-01,dcid:country/AUT,dcid:Count_BirthEvent,8545,P1M,dcs:Person +1975-01,dcid:country/AUT,dcid:Count_BirthEvent,8343,P1M,dcs:Person +1976-01,dcid:country/AUT,dcid:Count_BirthEvent,8047,P1M,dcs:Person +1977-01,dcid:country/AUT,dcid:Count_BirthEvent,7259,P1M,dcs:Person +1978-01,dcid:country/AUT,dcid:Count_BirthEvent,7157,P1M,dcs:Person +1979-01,dcid:country/AUT,dcid:Count_BirthEvent,7554,P1M,dcs:Person +1980-01,dcid:country/AUT,dcid:Count_BirthEvent,7481,P1M,dcs:Person +1981-01,dcid:country/AUT,dcid:Count_BirthEvent,7890,P1M,dcs:Person +1982-01,dcid:country/AUT,dcid:Count_BirthEvent,8142,P1M,dcs:Person +1983-01,dcid:country/AUT,dcid:Count_BirthEvent,7825,P1M,dcs:Person +1984-01,dcid:country/AUT,dcid:Count_BirthEvent,7424,P1M,dcs:Person +1985-01,dcid:country/AUT,dcid:Count_BirthEvent,7546,P1M,dcs:Person +1986-01,dcid:country/AUT,dcid:Count_BirthEvent,7468,P1M,dcs:Person +1987-01,dcid:country/AUT,dcid:Count_BirthEvent,7279,P1M,dcs:Person +1988-01,dcid:country/AUT,dcid:Count_BirthEvent,7442,P1M,dcs:Person +1989-01,dcid:country/AUT,dcid:Count_BirthEvent,7470,P1M,dcs:Person +1990-01,dcid:country/AUT,dcid:Count_BirthEvent,7612,P1M,dcs:Person +1991-01,dcid:country/AUT,dcid:Count_BirthEvent,7925,P1M,dcs:Person +1992-01,dcid:country/AUT,dcid:Count_BirthEvent,7932,P1M,dcs:Person +1993-01,dcid:country/AUT,dcid:Count_BirthEvent,8266,P1M,dcs:Person +1994-01,dcid:country/AUT,dcid:Count_BirthEvent,7798,P1M,dcs:Person +1995-01,dcid:country/AUT,dcid:Count_BirthEvent,7603,P1M,dcs:Person +1996-01,dcid:country/AUT,dcid:Count_BirthEvent,7436,P1M,dcs:Person +1997-01,dcid:country/AUT,dcid:Count_BirthEvent,7318,P1M,dcs:Person +1998-01,dcid:country/AUT,dcid:Count_BirthEvent,6860,P1M,dcs:Person +1999-01,dcid:country/AUT,dcid:Count_BirthEvent,6697,P1M,dcs:Person +2000-01,dcid:country/AUT,dcid:Count_BirthEvent,6587,P1M,dcs:Person +2001-01,dcid:country/AUT,dcid:Count_BirthEvent,6611,P1M,dcs:Person +2002-01,dcid:country/AUT,dcid:Count_BirthEvent,6594,P1M,dcs:Person +2003-01,dcid:country/AUT,dcid:Count_BirthEvent,6570,P1M,dcs:Person +2004-01,dcid:country/AUT,dcid:Count_BirthEvent,6780,P1M,dcs:Person +2005-01,dcid:country/AUT,dcid:Count_BirthEvent,6482,P1M,dcs:Person +2006-01,dcid:country/AUT,dcid:Count_BirthEvent,6526,P1M,dcs:Person +2007-01,dcid:country/AUT,dcid:Count_BirthEvent,6504,P1M,dcs:Person +2008-01,dcid:country/AUT,dcid:Count_BirthEvent,6739,P1M,dcs:Person +2009-01,dcid:country/AUT,dcid:Count_BirthEvent,6475,P1M,dcs:Person +2010-01,dcid:country/AUT,dcid:Count_BirthEvent,6414,P1M,dcs:Person +2011-01,dcid:country/AUT,dcid:Count_BirthEvent,6651,P1M,dcs:Person +2012-01,dcid:country/AUT,dcid:Count_BirthEvent,6389,P1M,dcs:Person +2013-01,dcid:country/AUT,dcid:Count_BirthEvent,6629,P1M,dcs:Person +2014-01,dcid:country/AUT,dcid:Count_BirthEvent,6666,P1M,dcs:Person +2015-01,dcid:country/AUT,dcid:Count_BirthEvent,6985,P1M,dcs:Person +2016-01,dcid:country/AUT,dcid:Count_BirthEvent,6980,P1M,dcs:Person +2017-01,dcid:country/AUT,dcid:Count_BirthEvent,7040,P1M,dcs:Person diff --git a/statvar_imports/eurostat/live_births_total_by_month/test_data/live_births_total_by_month_output.tmcf b/statvar_imports/eurostat/live_births_total_by_month/test_data/live_births_total_by_month_output.tmcf new file mode 100644 index 0000000000..39a28d1ba4 --- /dev/null +++ b/statvar_imports/eurostat/live_births_total_by_month/test_data/live_births_total_by_month_output.tmcf @@ -0,0 +1,8 @@ +Node: E:output->E0 +observationDate: C:output->observationDate +observationAbout: C:output->observationAbout +variableMeasured: C:output->variableMeasured +value: C:output->value +observationPeriod: C:output->observationPeriod +unit: C:output->unit +typeOf: dcs:StatVarObservation