Kalia, Packing Peanuts: The Role Synthetic Data Can Play in Enhancing Conventional Economic Prediction Models (2024-05)

Summary

In this May 2024 arXiv paper (arXiv:2405.07431v1), Vansh Murad Kalia, a candidate for a Master's of Arts in Quantitative Methods for the Social Sciences at Columbia University, argues that synthetic data, like packing peanuts around fragile items, can make economic prediction models perform better and more robustly when combined with real data. The study uses Affinity credit card spending and Womply small business datasets and reports substantial performance improvements from the hybrid approach over traditional economic modeling techniques, particularly where data is missing or limited. Prof. Gregory M. Eirich was the thesis advisor.

Summary drafted by a model from the document's text below and checked by script against that text before publication. It is a navigation aid, not a reading of what the document proves. Where AI is used

Cite this document

“Kalia, Packing Peanuts: The Role Synthetic Data Can Play in Enhancing Conventional Economic Prediction Models (2024-05).” Original at web.archive.org. Archived at Pandemic Darlings, https://pandemicdarlings.com/source-documents/kalia-packing-peanuts-the-role-synthetic-data-can-play-in-enhancing-co-61a5ada8a9/, SHA-256 5cc97e2f74ecc563b6f7ddb1251e25ae984dc655c3db801c14f75d430efd08e1.

Full text

Packing Peanuts:
The Role Synthetic Data Can Play in Enhancing

arXiv:2405.07431v1 [econ.GN] 13 May 2024
Conventional Economic Prediction Models
Vansh Murad Kalia

Candidate for Master’s of Arts in
Quantitative Methods for the Social Sciences

Columbia University

Thesis Advisor: Prof. Gregory M. Eirich

Abstract

Packing peanuts, as defined by Wikipedia, is a common loose-fill packag-
ing and cushioning material that helps prevent damage to fragile items.
In this paper, I propose that synthetic data, akin to packing peanuts,
can serve as a valuable asset for economic prediction models, enhancing
their performance and robustness when integrated with real data. This
hybrid approach proves particularly beneficial in scenarios where data is
either missing or limited in availability. Through the utilization of Affin-
ity credit card spending and Womply small business datasets, this study
demonstrates the substantial performance improvements achieved by em-
ploying a hybrid data approach, surpassing the capabilities of traditional
economic modeling techniques.

1
Index

Section Page

1. Introduction 3

2. Literature Review 3-5

3. Data 5

4. Methodology 6 - 12

4.1 Exploratory Data Analysis 6-7

4.2 Data Pre-processing 8

4.3 Model Selection 8 - 10

4.4 Model Testing Results 10 - 12

5. Conclusion 12

6. Limitations 13

7. Next Steps 13

References 14

2
1 Introduction
In recent years, the use of machine learning models for economic prediction
has gained significant traction. While the adoption of these techniques has
expedited the process of synthesizing vast amounts of data, one of the main
challenges that remains is obtaining the data itself (or enough of it, at least!).
Traditional approaches to data collection in the field of economics can be time-
consuming, expensive, and limited in scope. There are countless cases where
data is available but spotty, with missing samples. In such cases, synthetic data
has emerged as a promising candidate to help fill that gap, but what is synthetic
data?

On the highest level, synthetic data can be categorized into three main types:

• Derived from real datasets, inheriting their statistical properties.
• Generated independently of real data, without using any existing datasets.

• Hybrid in nature, combining aspects of the first two types.

This paper focuses on the Hybrid type, exploring its potential applications in
enhancing economic prediction models.

Utilizing data from Affinity and Womply, this study aims to investigate whether
the integration of synthetic data can improve model performance and robustness
in scenarios characterized by limited data availability, potentially outperforming
models reliant solely on real data.

2 Literature Review
Given the nascent nature of the academic intersection of economic prediction
models and synthetic data, there is not a lot of academic research that focuses
directly on this topic. As such, for this research, I leverage some academic lit-
erature on synthetic data in relational fields like computer science, to formulate
my hypothesis.

Synthetic Data Generation for Economists:1

In my search for academic literature at the intersection of synthetic data and
economics, this paper stands out as one of the most important contemporary
pieces. In this study, the authors address synthetic data generation within the
field of economics by recognizing the challenges associated with accessing and
handling sensitive or limited datasets. Koenecke and Varian discuss the method-
ologies and implications of generating synthetic data, providing economists with
1 Allison Koenecke and Hal Varian. Synthetic Data Generation for Economists. 2020.

arXiv: 2011.01374 [econ.GN].

3
a valuable resource for exploring and testing hypotheses in situations where real
data availability is constrained. The authors propose the use of synthetic data
as an alternative for economic researchers:

• Assist with privacy issues related to the use of data.
• Increase the number of samples available for a certain type of data.
• Test the robustness of existing models.

The paper contributes as an important piece to my research by offering insights
into the potential benefits of synthetic data and helping formulate my hypothesis
that using the hybrid of synthetic and real data should improve the performance
of an economic prediction model.

Macroeconomic Predictions using Payments Data and Machine Learn-
ing:2

In this study, the authors focus on predicting the economy’s short-term dynam-
ics and delve into economic forecasting by leveraging payments data and ma-
chine learning techniques. This paper aims to demonstrate that non-traditional
and timely data such as retail and wholesale payments, with the aid of nonlin-
ear machine learning approaches, can provide policymakers with sophisticated
models to accurately estimate key macroeconomic indicators in near real-time.
By incorporating advanced machine learning algorithms and non-linear learn-
ing approaches, Chapman and Desai show over 40 percent improved accuracy in
macroeconomic nowcasting. As a deeply quantitative study, this paper helped
me structure the quantitative analysis for my research and nudged me towards
the data I use as well.

Augmentation Techniques in Time Series Domain: A Survey and
Taxonomy3

This study offers a comprehensive overview of various data augmentation meth-
ods specifically tailored for time series data. In this paper, the authors delve
into a systematic classification of different augmentation techniques, categoriz-
ing them based on their underlying principles and applications. The authors
explore a wide array of augmentation approaches including traditional meth-
ods such as linear interpolation and synthetic data generation techniques like
Generative Adversarial Networks (GANs). They discuss the advantages, limita-
tions, and potential applications of each technique, providing insights into their
2 James T. E. Chapman and Ajit Desai. Macroeconomic Predictions using Payments Data

and Machine Learning. 2022. arXiv: 2209.00948 [econ.GN].
3 Guillermo Iglesias et al. “Data Augmentation techniques in time series domain: a survey

and taxonomy”. In: Neural Computing and Applications 35.14 (Mar. 2023), pp. 10123–10145.
issn: 1433-3058. doi: 10.1007/s00521- 023- 08459- 3. url: http://dx.doi.org/10.1007/
s00521-023-08459-3.

4
effectiveness in addressing various challenges encountered in time series anal-
ysis. Additionally, the paper examines the implications of data augmentation
on model generalization, robustness, and interpretability. Overall, this survey
and taxonomy have helped me navigate the landscape of data augmentation
techniques in the context of the time series analysis pertinent to my research.

K-Nearest Neighbor (k-NN) based Missing Data Imputation4

The authors of this paper explore the application of the K-Nearest Neighbor
(k-NN) algorithm for imputing missing data. They investigate the use of the
k-NN method as a means to address missing data in datasets and through their
research, they propose a framework that leverages the k-NN algorithm to predict
missing values based on the values of neighboring data points. This approach
aims to improve data completeness and accuracy in datasets affected by miss-
ing information. While this paper contributes to the field of data imputation
by offering a novel method that utilizes machine learning techniques to handle
missing data effectively, it’s not necessarily most suitable for my research as the
distance between the missing data points is too much to be able to efficiently
use the k-NN algorithm.

3 Data
For this research, I’ve opted to use the Affinity credit card spending datasets and
Womply small business datasets from the Economic Tracker database5 . These
datasets offer diverse features, but to narrow the focus for hypothesis testing,
attention is given to the daily spend 19 all variable from Affinity and the
merchants all variable from Womply. These variables can be described as
follows:

• daily spend 19 all: Daily spending in all merchant category codes (MCCs).
• merchants all: Percent change in the number of small businesses open,
calculated as a seven-day moving average, seasonally adjusted, and in-
dexed to January 4 to 31, 2020.

The daily spend 19 all variable comes from the Affinity dataset, as all spend-
ing features are measured relative to January 6 to February 2, 2020, seasonally
adjusted, and calculated as a seven-day moving average. There are additional
quartile features that are subdivisions by income using the median income of
the ZIP codes; q1 is the quartile with the lowest median income and q4 is the
quartile with the highest median income. I selected these variables on the high-
est level as they are ideal to test my hypothesis where there is missing data for
the merchants all that I would look to impute.
4 Della Murti et al. “K-Nearest Neighbor (K-NN) based Missing Data Imputation”. In:

Oct. 2019, pp. 83–88. doi: 10.1109/ICSITech46713.2019.8987530.
5 https://github.com/OpportunityInsights/EconomicTracker

5
4 Methodology
To test my hypothesis, I will create a real-life example using this dataset and my
aim will be to create the best possible model to predict spending (daily spend 19 all)
using the independent variable (merchants all)

4.1 Exploratory Data Analysis
As an initial step in exploratory data analysis, I examine the descriptive statis-
tics of the original dataset:

Table 1: Descriptive Statistics

daily spend 19 all merchants all
count 1253.000 109.000
mean 0.280 -0.056
std 0.267 0.067
min -0.643 -0.302
25% 0.124 -0.066
50% 0.243 -0.049
75% 0.455 -0.021
max 1.200 0.086

The descriptive statistics provide basic insights into the dataset. However, they
do not offer much relevant information for addressing the research question.
Thus, I proceed towards exploratory data analysis to examine the temporal
distribution of the two variables.

Figure 1: Data Distribution for Daily Spend

6
Figure 2: Data Distribution for All Merchants

Figure 3: Missing Data Across Variables

Both Figure 1 and Figure 2 reveal notable trends, particularly a substantial
drop in both variables during 2020, coinciding with the onset of the COVID-19
pandemic. Additionally, there are intriguing outliers, such as the end of end-of-
year data points in Figure 1. Figure 3 reveals that there are no missing variables
for Daily Spend features but there are a lot of missing data for merchants all.
Other than this, not much can be derived from an Exploratory Data Analysis
that is relevant to my research question. The missing data will be addressed
during preprocessing to ensure the success of this real-life example.

7
4.2 Data Pre-processing
Despite the well-organized and clean nature of the data, several challenges exist,
including datatype mismatches, DateTime information, filtering, and missing
variables. To address these issues, I implement various data pre-processing tech-
niques, such as handling different data types, managing DateTime information,
and applying filtering. However, due to the density of the data, a careful selec-
tion of features is necessary for visualization purposes. Most notably, Womply’s
business data is provided on a weekly basis, in contrast to the daily spending
data. This missing data could potentially affect the accuracy of my model’s
predictions and to address this discrepancy, I plan to generate synthetic data
to fill this gap and facilitate meaningful comparisons with non-synthetic data
models.

I create four base datasets that cover conventional methods of missing data im-
putation used in Economics which will then be compared with the fifth model
trained on the hybrid dataset built from generated synthetic data and real data.
The techniques I follow are removing missing rows, global mean imputation, and
Monte Carlo simulations and the base models for testing will be generated on
the below datasets:

• Original dataset with no imputations
• Original dataset with missing rows removed
• Mean-imputed dataset that fills missing values using global mean

• Monte Carlo simulations imputed dataset

As mentioned in the literature review, I also considered the k-Nearest Neighbors
(k-NN) technique for another base model. However, it proved unsuitable for
the data I’m dealing with due to the lack of neighboring data points for proper
imputation. These base datasets will facilitate the creation of the first four base
models for testing and evaluation.

4.3 Model Selection
As I begin the model selection process for testing and evaluation, it is impor-
tant to recognize that this research involves a variety of complications with using
time-series economic data, including missing data for a variable of interest and
endogeneity concerns. As such, choosing the right model and evaluation tech-
niques is of immense importance.

OLS Regression:

In the initial stage of analysis, I utilize Ordinary Least Squares (OLS) regres-
sion to investigate the linear relationship between the variables of interest. OLS

8
regression is a widely used statistical method for estimating the relationship be-
tween a dependent variable and one or more independent variables by minimiz-
ing the sum of the squared differences between observed and predicted values6 .
By fitting a linear regression model to the data, I aim to identify any significant
linear associations and quantify the strength and direction of these relationships.
Additionally, OLS regression provides insights into the relative importance of
each independent variable in explaining the variation observed in the dependent
variable. This initial analysis will help inform subsequent modeling approaches
and provide valuable insights into the underlying factors influencing the target
variable’s behavior.

Random Forest Model:

Beyond capturing linear relationships, I employ a Random Forest model as
a secondary economic prediction model. Random Forest is an ensemble learn-
ing method that constructs multiple decision trees during training and outputs
the class that is the mode of the classes (classification) or mean prediction (re-
gression) of the individual trees7 . It is a powerful tool for capturing nonlinear
relationships and has been widely applied in economic research for variable se-
lection, forecasting, and causal inference8 . The Random Forest algorithm is
well-suited for handling complex relationships and interactions within the data,
providing a more comprehensive understanding of the factors influencing the
outcome. As such, I also use a Random Forest Model to compare the model
performance across all the datasets.

Synthetic Data Generation:

To complete the Model Selection process, I will now outline the unique approach
I take to generate synthetic data to fill data gaps within the Womply business
dataset. I train a Random Forest Model on the second dataset mentioned in the
pre-processing section (original dataset with missing rows removed) as it repre-
sents the cleanest form of real data. While the original dataset contains missing
daily values for merchants all variable, it still has enough weekly values for
me to be able to use it as a target variable to train the random forest model.
Then, I use the trained model to predict (or impute) the missing values within
the original dataset.

This approach leverages Affinity data’s daily spend 19 all features as in-
6 Bartosz Zdaniuk. “Ordinary Least-Squares (OLS) Model”. In: Encyclopedia of Quality

of Life and Well-Being Research. Ed. by Alex C. Michalos. Dordrecht: Springer, 2014. doi:
10.1007/978- 94- 007- 0753- 5\_2008. url: https://doi.org/10.1007/978- 94- 007- 0753-
5%5C_2008.
7 Hector M. Tripp et al. “DiabetIA: a real-world research database to predict de novo

diabetic complications using artificial intelligence”. In: Nature Diabetes 4.3 (2023), pp. 201–
211. doi: 10.1038/s42255-023-00712-9.
8 Philippe Goulet Coulombe. “The Macroeconomy as a Random Forest”. In: arXiv preprint

arXiv:2006.12724 (2020).

9
puts and learns any non-linear relationships between the target variable and
the features. By doing so, it enhances the accuracy of filling the gaps in the
merchants all column, leading to a more robust model. This leads to the cre-
ation of the fifth dataset for comparison against all the base models, and any
model trained on this hybrid dataset will be referred to as Model 5 from here
onwards.

Below is a scatter plot of the distribution of this hybrid dataset:

Figure 4: Hybrid Data Distribution for All Merchants

In comparison, below is a scatter plot of the distribution of just the real data:

Figure 5: Hybrid Data Distribution for All Merchants

10
4.4 Model Testing Results
Now that all of the models are ready, let’s take a look at how Model 5 (the
comparison model) performs against all the base models across OLS Regression
and Random Forest Models.

Base Models OLS Regression Results

Model 1 coef std err t P> |t| [0.025 0.975]
const 0.0582*** 0.017 3.369 0.001 0.024 0.092
merchants all 1.6710*** 0.198 8.430 0.000 1.278 2.064

Model 2 coef std err t P> |t| [0.025 0.975]
const 0.0582*** 0.017 3.369 0.001 0.024 0.092
merchants all 1.6710*** 0.198 8.430 0.000 1.278 2.064

Model 3 coef std err t P> |t| [0.025 0.975]
const 0.3737*** 0.023 16.473 0.000 0.329 0.418
merchants all 1.6710*** 0.381 4.382 0.000 0.923 2.419

Model 4 coef std err t P> |t| [0.025 0.975]
const 0.0582*** 0.017 3.369 0.001 0.024 0.092
merchants all 1.6710*** 0.198 8.430 0.000 1.278 2.064

Model 5 coef std err t P> |t| [0.025 0.975]
const 0.3302*** 0.006 51.379 0.000 0.318 0.343
merchants all 4.2133*** 0.165 25.588 0.000 3.890 4.536

From the above table, it’s clear that while all the models have statistically sig-
nificant coefficients and constants, Model 5 stands out in comparison to the
baseline models with its notably higher coefficient value (4.2133) for the vari-
able merchants all, indicating a stronger impact on the dependent variable.
Moreover, Model 5 exhibits lower standard errors for both the constant and
merchants all, suggesting greater precision in the coefficient estimates. The
high t-values (51.379 and 25.588 respectively) and extremely low p-values indi-
cate high significance, further supporting the robustness of Model 5. Overall,
Model 5 appears to offer a more accurate and statistically significant represen-
tation of the relationship between the variables compared to the other models.
This result confirms my hypothesis, however, I will still look to substantiate it
using Random Forest Models and see how Model 5 compares to the baseline
models.

11
Table 2: Random Forest Model Results

Model Average MAE Average MSE Average R-squared
1 NA NA NA
2 0.162 0.042 -5.92
3 0.217 0.077 -0.75
4 0.232 0.088 -1.06
5 0.092 0.017 0.55

Similarly to the OLS Regression results, Table 2 demonstrates Model 5’s supe-
rior performance as it exhibits the lowest average Mean Absolute Error (MAE)
of 0.092 and the lowest average Mean Squared Error (MSE) of 0.017, indicating
the closest proximity of predicted values to the actual values compared to other
models. Additionally, Model 5 achieves the highest average R-squared value of
0.55, suggesting that it explains a higher proportion of the variance in the de-
pendent variable. It should be noted that Model 1 is marked NA as the dataset
for Model 1 has missing values and is not suitable for a Random Forest analysis.
Models 3 and 4 display poorer performance across all metrics while Model 2,
although showing a low average MAE and MSE, has a substantially negative
R-squared value, indicating poor model fit or potential overfitting. Therefore,
Model 5 emerges as the most favorable choice among the presented Random
Forest models, demonstrating superior predictive accuracy and model robust-
ness.

5 Conclusion
I started this research to explore whether the integration of Synthetic Data
could enhance model performance and robustness in scenarios characterized by
limited data availability. Based on the literature review, I hypothesized that
employing the hybrid approach of synthetic and real data should improve the
performance of an economic prediction model, surpassing the efficacy of utiliz-
ing only real data. To test this hypothesis, I set up a real-life example using
the Affinity and Womply datasets and created four different baseline models
covering the conventional data-handling techniques used in economic prediction
modeling. These techniques covered using the original dataset with no imputa-
tions, the original dataset removing rows with missing data, imputing missing
values with global mean, and Monte Carlo simulations. My comparison model
was trained on a dataset created using an advanced data augmentation tech-
nique that leverages Random Forest Models to generate Synthetic Data and
use it in conjunction with real data. The comparison model outperformed all
the baseline models across both OLS Regression testing and Random Forest
Modeling, giving me strong conviction that my hypothesis is correct.

12
6 Limitations
In terms of limitations of this paper, there were quite a few that I faced through-
out the course of the research:

• As highlighted in the literature review, the nascent nature of this topic
meant that there was not a lot of reliable academic research I could lever-
age that focused directly on the intersection of economics and synthetic
data. I consider this a huge limitation, as a lot more literature on the
topic could have changed the structure of my research.
• The dataset I have been working with had a very high number of missing
values for the target variable which could’ve easily led to an imbalanced
dataset, adding bias in the generated data. If I had access to more data,
or data with more frequency, I potentially could’ve used other baseline
modeling techniques like k-NN and seen different results.
• Another limitation would be lacking the skills required to build more so-
phisticated models like Generative Adversarial Networks (GANs) or Vari-
ational Auto-Encoders (VAEs). Based on the literature review, it is very
likely that synthetic data generated through either of these models would
perform better than synthetic data generated by a Random Forest Model.

Even considering these limitations, I believe this research is well grounded in
both qualitative and quantitative logic proving valid grounds for my hypothesis
to be correct.

7 Next Steps
In terms of next steps, I would like to create three more hybrid datasets using
the following techniques that I discovered through my literature review9 and
include them as comparison models to the existing tests:

• A library like Datawig to leverage Deep Learning Neural Networks.
• Generative Adversarial Network (GAN) to generate more accurate syn-
thetic data.
• Variational Auto-Encoder (VAE) to generate more accurate synthetic data
by accounting for variance in time-series data.

All of these are more sophisticated modeling techniques that have the potential
to produce more robust results compared to Random Forest Models and are
something I can look forward to implementing in economic prediction models.
9 Guillermo Iglesias et al. “Data Augmentation techniques in time series domain: a survey

and taxonomy”. In: Neural Computing and Applications 35.14 (Mar. 2023), pp. 10123–10145.
issn: 1433-3058. doi: 10.1007/s00521- 023- 08459- 3. url: http://dx.doi.org/10.1007/
s00521-023-08459-3.

13
References
[1] James T. E. Chapman and Ajit Desai. Macroeconomic Predictions using
Payments Data and Machine Learning. 2022. arXiv: 2209.00948 [econ.GN].
[2] Philippe Goulet Coulombe. “The Macroeconomy as a Random Forest”. In:
arXiv preprint arXiv:2006.12724 (2020).
[3] Guillermo Iglesias et al. “Data Augmentation techniques in time series do-
main: a survey and taxonomy”. In: Neural Computing and Applications
35.14 (Mar. 2023), pp. 10123–10145. issn: 1433-3058. doi: 10.1007/s00521-
023-08459-3. url: http://dx.doi.org/10.1007/s00521-023-08459-3.
[4] Allison Koenecke and Hal Varian. Synthetic Data Generation for Economists.
2020. arXiv: 2011.01374 [econ.GN].
[5] Della Murti et al. “K-Nearest Neighbor (K-NN) based Missing Data Im-
putation”. In: Oct. 2019, pp. 83–88. doi: 10.1109/ICSITech46713.2019.
8987530.
[6] Hector M. Tripp et al. “DiabetIA: a real-world research database to pre-
dict de novo diabetic complications using artificial intelligence”. In: Nature
Diabetes 4.3 (2023), pp. 201–211. doi: 10.1038/s42255-023-00712-9.
[7] Bartosz Zdaniuk. “Ordinary Least-Squares (OLS) Model”. In: Encyclope-
dia of Quality of Life and Well-Being Research. Ed. by Alex C. Michalos.
Dordrecht: Springer, 2014. doi: 10.1007/978- 94- 007- 0753- 5\_2008.
url: https://doi.org/10.1007/978-94-007-0753-5%5C_2008.

File and source

File: kalia-2024-arxiv-2405-07431-packing-peanuts-synthetic-data.pdf
Size: 587,389 bytes
SHA-256: 5cc97e2f74ecc563b6f7ddb1251e25ae984dc655c3db801c14f75d430efd08e1
Our copy: kalia-2024-arxiv-2405-07431-packing-peanuts-synthetic-data.pdf
Original: web.archive.org