The objective is to unify an inconsistently coded categorical variable
in a panel dataset according to a mapping (transition) table.
The mapping (transition) table is the core element of the process.
The function has a modular design with three arguments `data`, `mappings`, and `ml`. Each
of these arguments is of a `list` type, wherein the
`ml` argument is optional. Arguments are separated to
identify the core elements of the `cat2cat` procedure.
Although this function seems
complex initially, it is built to offer a wide range of
applications for complex tasks. The function contains
many validation checks to prevent incorrect usage.
The function has to be applied iteratively for each two neighboring periods
of a panel dataset.
The prune_c2c function could be needed to limit growing number
of replications.
cat2cat(
data = list(old = NULL, new = NULL, time_var = NULL, cat_var = NULL, cat_var_old =
NULL, cat_var_new = NULL, id_var = NULL, multiplier_var = NULL),
mappings = list(trans = NULL, direction = NULL, freqs_df = NULL),
ml = list(data = NULL, cat_var = NULL, method = NULL, features = NULL, args = NULL)
)`named list` with fields `old`, `new`, `cat_var` (or `cat_var_old` and `cat_var_new`), `time_var` and optional `id_var`,`multiplier_var`.
`named list` with 3 fields `trans`, `direction` and optional `freqs_df`.
`named list` (optional) with up to 5 fields `data`, `cat_var`, `method`, `features` and optional `args`.
`named list` with 2 fields old and new - 2 data.frames. There will be added additional columns like index_c2c, g_new_c2c, wei_freq_c2c, rep_c2c, wei_(ml method name)_c2c. Additional columns will be informative only for a one data.frame as we always make the changes to one direction. The new columns are added instead of the additional metadata as we are working with new datasets where observations could be replicated. For the transparency the probability and number of replications are part of each observation in the `data.frame`.
data args
data.frame older time point in a panel
data.frame more recent time point in a panel
character(1) name of the time variable.
character(1) name of the categorical variable.
Optional character(1) name of the categorical variable in the older time point. Default `cat_var`.
Optional character(1) name of the categorical variable in the newer time point. Default `cat_var`.
Optional character(1) name of the unique identifier variable - if this is specified then for subjects observed in both periods, the direct mapping is applied.
Optional character(1) name of the multiplier variable - number of replication needed to reproduce the population
Only for the backward compatibility check the definition in the description of the mappings argument
mappings args
data.frame with 2 columns - mapping (transition) table - all categories for cat_var in old and new datasets have to be included. First column contains an old encoding and second a new one. The mapping (transition) table should to have a candidate for each category from the targeted for an update period.
character(1) direction - "backward" or "forward"
Optional - data.frame with 2 columns where first one is category name (base period) and second counts. If It is not provided then is assessed automatically. Artificial counts for each variable level in the base period. It is optional nevertheless will be often needed, as gives more control. It will be used to assess the probabilities. The multiplier variable is omitted so sb has to apply it in this table.
Optional ml args
data.frame - dataset with features and the `cat_var`.
character(1) - the dependent variable name.
character vector - one or a few from "knn", "rf" and "lda" methods - "knn" k-NearestNeighbors, "lda" Linear Discrimination Analysis, "rf" Random Forest
character vector of features names where all have to be numeric or logical
optional - list parameters: knn: k ; rf: ntree
Without ml section only simple frequencies are assessed. When ml model is broken then weights from simple frequencies are taken. `knn` method is recommended for smaller datasets.
`trans` arg columns and the `cat_var` column have to be of the same type. The mapping (transition) table should to have a candidate for each category from the targeted for an update period. The observation from targeted for an updated period without a matched category from base period is removed. If you want to leave NA values add `c(NA, NA)` row to the `trans` table. Please check the vignette for more information.
if (FALSE) {
data("occup_small", package = "cat2cat")
data("occup", package = "cat2cat")
data("trans", package = "cat2cat")
occup_old <- occup_small[occup_small$year == 2008, ]
occup_new <- occup_small[occup_small$year == 2010, ]
# Adding the dummy level to the mapping table for levels without a candidate
# The best to fill them manually with proper candidates, if possible
# In this case it is only needed for forward mapping, to suppress warnings
trans2 <- rbind(
trans,
data.frame(
old = "no_cat",
new = setdiff(c(occup_new$code), trans$new)
)
)
# default only simple frequencies
occup_simple <- cat2cat(
data = list(
old = occup_old, new = occup_new, cat_var = "code", time_var = "year"
),
mappings = list(trans = trans2, direction = "forward")
)
mappings <- list(trans = trans, direction = "backward")
ml_setup <- list(
data = occup_small[occup_small$year >= 2010, ],
cat_var = "code",
method = "knn",
features = c("age", "sex", "edu", "exp", "parttime", "salary"),
args = list(k = 10)
)
# ml model performance check
print(cat2cat_ml_run(mappings, ml_setup))
# additional probabilities from knn
occup_ml <- cat2cat(
data = list(
old = occup_old, new = occup_new, cat_var = "code", time_var = "year"
),
mappings = mappings,
ml = ml_setup
)
}