Estimating heterogeneous panels with csdm
Joao Claudio Macosso
Source:vignettes/introduction_to_csdm.Rmd
introduction_to_csdm.RmdScope
csdm estimates heterogeneous panel models in which
unobserved common factors may generate dependence across units. The
package implements:
- mean group (MG) estimation (Pesaran & Smith, 1995);
- static common correlated effects (CCE) (Pesaran, 2006);
- dynamic CCE (DCCE) (Chudik & Pesaran, 2015); and
- a cross-sectionally augmented ARDL fit with implied adjustment and long-run parameters.
These estimators share a mean-group structure: a separate regression is fitted for every eligible unit and the unit-level coefficients are averaged. CCE-based models augment those regressions with cross-sectional averages that proxy the latent common-factor space.
The package adopts parts of the model structure used by Stata’s
xtdcce2 (Ditzen, 2018), but
it does not claim complete command or option parity. Pooled
restrictions, estimation weights, CS-DL, CS-ECM, alternative fit-level
covariance estimators, and prediction on new data are currently
unavailable.
Data and sample
The bundled PWT_60_07 data contain 93 countries observed
annually from 1960 through 2007. The variables follow the original
xtdcce2 example:
-
log_rgdpo: log real output; -
log_hc: log human capital; -
log_ck: log physical capital; and -
log_ngd: log population growth plus a 5% break-even investment rate.
The 93 values of log_ngd in 1960 are missing because its
construction uses a growth rate. The worked example starts in 1970 and
uses 15 countries to keep the vignette quick to build.
data(PWT_60_07, package = "csdm")
keep_ids <- unique(PWT_60_07$id)[1:15]
dat <- subset(PWT_60_07, id %in% keep_ids & year >= 1970)
dim(dat)
#> [1] 570 6
range(dat$year)
#> [1] 1970 2007The examples below are levels regressions. Calling them growth regressions would require a differenced dependent variable, which is not part of the formula used here.
Estimators
Let index units and index time. A heterogeneous panel model is
Mean group
MG estimates the equation separately for each unit and averages the identified unit coefficients:
where is the common set of eligible units and is its size. The reported covariance is the cross-unit sample covariance of the coefficient vectors divided by . Inference uses a large- normal approximation.
MG permits slope heterogeneity but does not model common-factor dependence.
Common correlated effects
CCE augments each unit equation with cross-sectional averages:
where commonly contains the averages of the dependent variable and regressors. Under the CCE assumptions, these averages span the relevant common-factor space and allow consistent estimation of the unit slopes (Pesaran, 2006). Adding averages does not by itself guarantee that the factor space is adequately represented; rank, dimensions, and the choice of averages still matter.
Dynamic CCE
DCCE adds dynamics and lags of the cross-sectional averages:
csdm_lr(type = "ardl", ylags = P, xdlags = Q) controls
model lags, and csdm_csa(lags = S) controls lags of the
averages. Dynamic mean-group estimates can have
short-
bias; CCE augmentation does not remove that bias.
CS-ARDL output
model = "cs_ardl" fits the same unit-level levels ARDL
used by DCCE and then transforms its coefficients. For unit
,
define
The package reports as the adjustment coefficient and as the long-run ratio before computing their mean-group summaries. Ratios are undefined when is numerically zero. The implementation also stores AR-root diagnostics, but it does not automatically discard unstable units.
These transformations do not fit a separate error-correction model
and do not establish cointegration. The levels component is
the fitted levels ARDL coefficient vector, rather than the complete
transformed short-run ECM vector.
Specifying and fitting models
The formula contains the contemporaneous economic regressors. Two specification objects add CCE terms and dynamics:
-
csdm_csa(vars, lags)selects variables whose cross-sectional averages enter the model and their maximum lags; -
csdm_lr(type, ylags, xdlags)selects lags of the dependent variable and regressors.
With vars = "_all", averages are constructed from the
evaluated response and economic model-matrix columns, excluding
intercepts and unit trends. An explicit character vector instead refers
to numeric columns in data. vars = "_none"
turns off CCE augmentation where the chosen estimator permits it.
form <- log_rgdpo ~ log_hc + log_ck + log_ngd
csa_vars <- c("log_rgdpo", "log_hc", "log_ck", "log_ngd")
static_csa <- csdm_csa(vars = csa_vars)
dynamic_csa <- csdm_csa(vars = csa_vars, lags = 3)
ardl_1_0 <- csdm_lr(type = "ardl", ylags = 1, xdlags = 0)MG
fit_mg <- csdm(
form, data = dat, id = "id", time = "year", model = "mg"
)
summary(fit_mg)
#> csdm summary: Mean Group Model (MG)
#> Formula: log_rgdpo ~ log_hc + log_ck + log_ngd
#> N: 15, T: 38
#> Number of obs: 570
#> R-squared (mg): 0.9449
#> CD = 3.2379, p = 0.0012
#> (For additional CD diagnostics, use cd_test())
#>
#> Mean Group:
#> Coef. Std. Err. z P>|z| Signif. CI 2.5% CI 97.5%
#> (Intercept) 6.5609 0.8648 7.5867 0.0000 *** 4.8659 8.2558
#> log_hc 0.1725 0.8530 0.2022 0.8398 -1.4993 1.8443
#> log_ck 0.3056 0.1359 2.2485 0.0245 * 0.0392 0.5720
#> log_ngd 0.7800 0.3777 2.0652 0.0389 * 0.0398 1.5203
#>
#> Mean Group Variables: log_hc, log_ck, log_ngd
#> Cross Sectional Averaged Variables: none (lags=0)
#>
#> Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1CCE
fit_cce <- csdm(
form, data = dat, id = "id", time = "year", model = "cce",
csa = static_csa
)
summary(fit_cce)
#> csdm summary: Static Common Correlated Error Model (CCE)
#> Formula: log_rgdpo ~ log_hc + log_ck + log_ngd
#> N: 15, T: 38
#> Number of obs: 570
#> R-squared (mg): 0.9777
#> CD = -2.6758, p = 0.0075
#> (For additional CD diagnostics, use cd_test())
#>
#> Mean Group:
#> Coef. Std. Err. z P>|z| Signif. CI 2.5% CI 97.5%
#> (Intercept) 1.9003 2.1195 0.8965 0.3700 -2.2540 6.0545
#> log_hc -1.4921 1.0152 -1.4697 0.1416 -3.4819 0.4977
#> log_ck 0.1367 0.0956 1.4298 0.1528 -0.0507 0.3240
#> log_ngd 0.8075 0.2972 2.7175 0.0066 ** 0.2251 1.3899
#>
#> Mean Group Variables: log_hc, log_ck, log_ngd
#> Cross Sectional Averaged Variables: log_rgdpo, log_hc, log_ck, log_ngd (lags=0)
#>
#> Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1DCCE
fit_dcce <- csdm(
form, data = dat, id = "id", time = "year", model = "dcce",
csa = dynamic_csa,
lr = ardl_1_0
)
summary(fit_dcce)
#> csdm summary: Dynamic Common Correlated Error Model (DCCE)
#> Formula: log_rgdpo ~ log_hc + log_ck + log_ngd
#> N: 15, T: 38
#> Number of obs: 525
#> R-squared (mg): 0.9861
#> CD = -2.392, p = 0.0168
#> (For additional CD diagnostics, use cd_test())
#>
#> Mean Group:
#> Coef. Std. Err. z P>|z| Signif. CI 2.5% CI 97.5%
#> (Intercept) 9.2888 8.0386 1.1555 0.2479 -6.4666 25.0441
#> log_hc -1.9558 1.5659 -1.2490 0.2117 -5.0249 1.1132
#> log_ck 0.6666 0.2424 2.7499 0.0060 ** 0.1915 1.1417
#> log_ngd -0.3178 1.1271 -0.2819 0.7780 -2.5268 1.8913
#> lag1_log_rgdpo -0.0745 0.0555 -1.3424 0.1795 -0.1834 0.0343
#>
#> Mean Group Variables: log_hc, log_ck, log_ngd, lag1_log_rgdpo
#> Cross Sectional Averaged Variables: log_rgdpo, log_hc, log_ck, log_ngd (lags=3)
#>
#> Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1CS-ARDL
fit_cs_ardl <- csdm(
form, data = dat, id = "id", time = "year", model = "cs_ardl",
csa = dynamic_csa,
lr = ardl_1_0
)
summary(fit_cs_ardl)
#> csdm summary: Cross-Sectional ARDL (CS-ARDL)
#> Formula: log_rgdpo ~ log_hc + log_ck + log_ngd
#> N: 15, T: 38
#> Number of obs: 525
#> R-squared (mg): 0.9861
#>
#> CD = -2.392, p = 0.0168
#> (For additional CD diagnostics, use cd_test())
#>
#> Levels ARDL Est.
#> Coef. Std. Err. z P>|z| Signif. CI 2.5% CI 97.5%
#> (Intercept) 9.2888 8.0386 1.1555 0.2479 -6.4666 25.0441
#> log_hc -1.9558 1.5659 -1.2490 0.2117 -5.0249 1.1132
#> log_ck 0.6666 0.2424 2.7499 0.0060 ** 0.1915 1.1417
#> log_ngd -0.3178 1.1271 -0.2819 0.7780 -2.5268 1.8913
#> lag1_log_rgdpo -0.0745 0.0555 -1.3424 0.1795 -0.1834 0.0343
#>
#> Adjust. Term
#> Coef. Std. Err. z P>|z| Signif. CI 2.5% CI 97.5%
#> lr_log_rgdpo -1.0745 0.0555 -19.3513 0 *** -1.1834 -0.9657
#>
#> Long Run Est.
#> Coef. Std. Err. z P>|z| Signif. CI 2.5% CI 97.5% n_used
#> lr_log_hc -1.8378 1.4743 -1.2465 0.2126 -4.7274 1.0518 15
#> lr_log_ck 0.6106 0.2076 2.9409 0.0033 ** 0.2037 1.0175 15
#> lr_log_ngd -0.4098 1.2364 -0.3314 0.7403 -2.8330 2.0134 15
#>
#> Mean Group Variables: lag1_log_rgdpo, log_hc, log_ck, log_ngd
#> Cross Sectional Averaged Variables: log_rgdpo, log_hc, log_ck, log_ngd (lags=3)
#> Long Run Variables: log_hc, log_ck, log_ngd
#> Cointegration variable(s): log_rgdpo
#>
#> Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
coef(fit_cs_ardl, component = "long_run")
#> lr_log_hc lr_log_ck lr_log_ngd
#> -1.8377835 0.6105883 -0.4097798The four calls use the same levels formula, so their coefficient meanings are directly comparable only after accounting for their different dynamic and CCE terms. In particular, a contemporaneous coefficient in DCCE is not a long-run effect.
Samples, missing values, and identification
csdm() requires unique, nonmissing unit-time keys. By
default, numeric time indexes advance in steps of one; set
time_step when the intended grid differs. Lags respect gaps
rather than treating the previous observed row as the previous
period.
subset is evaluated before model construction. The
supported missing-value policies are na.omit,
na.exclude, and na.fail. Unit regressions must
retain positive residual degrees of freedom, and every economic
coefficient must be identified after projecting out the CCE terms. Units
that fail these checks are listed with reasons in
fit$units. At least two eligible units are required.
For csa = csdm_csa("_all"), the source sample for
cross-sectional averages is the complete base formula sample after
subsetting and before dynamic lag trimming. It therefore need not equal
the final set of fitted observations. Set fullsample = TRUE
to calculate each average from all finite observations of that variable
in the selected sample. This is useful when the averaging variables have
different missing-value patterns and mirrors the fullsample
option used in xtdcce2.
Inference and R model methods
Standard methods expose the fitted model and its sample:
coef(fit_cce)
#> (Intercept) log_hc log_ck log_ngd
#> 1.9002762 -1.4921059 0.1366750 0.8075283
sqrt(diag(vcov(fit_cce)))
#> (Intercept) log_hc log_ck log_ngd
#> 2.11954323 1.01523715 0.09559111 0.29715332
nobs(fit_cce)
#> [1] 570
head(model.frame(fit_cce))
#> log_rgdpo log_hc log_ck log_ngd
#> 1 8.016315 0.7679234 11.79632 -2.727981
#> 2 8.035557 0.7790388 11.84772 -2.712420
#> 3 8.035343 0.7901542 11.90875 -2.702157
#> 4 8.060844 0.8012696 11.97760 -2.697793
#> 5 8.075463 0.8123851 12.03988 -2.701199
#> 6 8.034805 0.8235005 12.07300 -2.709801
head(residuals(fit_cce, format = "long"))
#> id year .row residual
#> 11 1 1970 1 0.01979243
#> 12 1 1971 2 -0.00607203
#> 13 1 1972 3 -0.03194126
#> 14 1 1973 4 -0.03727097
#> 15 1 1974 5 0.00548517
#> 16 1 1975 6 0.02278676
head(fitted(fit_cce, format = "vector"))
#> 11 12 13 14 15 16
#> 7.996523 8.041629 8.067284 8.098115 8.069978 8.012019For CS-ARDL, coef() and vcov() accept
component = "levels", "adjustment",
"long_run", or "all". Every component uses a
common eligible-unit sample for its coefficients and covariance. Exact
algebraic relationships can make the combined "all"
covariance singular.
The package also supplies tidy(), glance(),
and augment() methods:
broom::tidy(fit_cce, conf.int = TRUE)
broom::glance(fit_cce)
broom::augment(fit_cce)
modelsummary::modelsummary(
list(MG = fit_mg, CCE = fit_cce, DCCE = fit_dcce),
statistic = "std.error"
)Cross-sectional dependence in residuals and variables
Let
denote the fitted residual and
the number of overlapping finite observations for units
and
.
The classical statistic implemented by cd_test() is
It uses pairwise-complete correlations by default. Large absolute values are evidence against the null represented by the standard-normal reference approximation. Depending on the theoretical formulation, that null is stated as cross-sectional independence or sufficiently weak dependence (Pesaran, 2015, 2021).
For a balanced panel, CDw draws one independent Rademacher weight per unit, independently of the data. Write for the input demeaned within unit. Equation (30) of the published Juodis-Reese paper (Juodis & Reese, 2022, p. 1197) is
The paper denotes the residuals by
;
its unit intercepts make their time means zero. The within-unit
demeaning above also allows the package to accept raw variables. Since
,
is the pooled mean squared demeaned input. The inverse pooled variance
is essential: CDw uses weighted covariances with this common scale,
rather than individually standardized correlations. The shorthand
used elsewhere denotes the same unordered pairs as the displayed double
sum. Set seed to reproduce the random weights without
changing the caller’s random-number state.
With reps = G, the test draws
independent sets of unit weights and combines the resulting statistics
using equation (33) of Juodis and Reese:
The default reps = 1 preserves the single-draw
statistic. The authors suggest keeping the number of draws modest, for
example 30, since increasing
can amplify lower-order terms.
CDw+ follows equation (32) of Juodis and Reese, which applies the power-enhancement principle of Fan, Liao, and Yao (Fan et al., 2015):
The threshold is applied to the ordinary residual correlation, without a multiplier. The enhancement is nonnegative, so CDw+ is not a second independent random-sign test. With multiple draws, CDw+ adds this screening term once to .
CD* uses the Pesaran-Xie correction algebra (Pesaran & Xie, 2021, 2026), with a
standardized-PCA implementation variant. For matrix and data inputs,
write
,
where
is the unit’s sample standard deviation. Extract n_pc
principal components from
,
normalizing the loading matrix so that
.
Set
and
.
The implemented correction is
Pesaran-Xie estimate the PCs from unstandardized observations.
Standardizing before PCA can change the factor space when unit variances
differ, so this package’s variant should not be described as an exact
reproduction of their PCA procedure. Setting n_pc = 0
returns classical CD on the supplied panel. The correction requires a
nondegenerate denominator; a numerically computable result alone does
not establish that the asymptotic assumptions are suitable.
For fitted models, CD* starts from the partial residual
,
retaining the fitted common-factor component. Here
includes the economic regressors and any intercept, unit trend, and
constructed economic lags. The CSA contribution is retained for PCA,
following the input construction in Pesaran-Xie’s regression procedure.
CD, CDw, and CDw+ instead use the full fitted residuals. Passing
residuals(fit) directly to CD* therefore tests a different
input from passing the fit itself. CD* records its input and sample
exclusions in tests$CDstar; in an "all"
result, its sample dimensions are in tests$CDstar$N and
tests$CDstar$T.
All reported p-values are two-sided standard-normal approximations,
.
The implemented tests do not estimate a serial-correlation correction or
establish validity for nonstationary series. The papers impose
time-series, factor, and panel-dimension conditions; accepting a numeric
variable does not establish those conditions. In particular, classical
CD can be biased on CCE residuals after estimating common time
parameters. CDw addresses that problem under Juodis-Reese’s assumptions,
including
.
Omitting pairs or using very short overlaps can also affect classical CD
calibration; min_overlap = 2 is a computational
minimum.
cd_test(fit_mg, type = "CD")
#> Cross-sectional dependence tests
#> N = 15, T = 38
#>
#> statistic p.value
#> CD 3.238 0.001
cd_test(fit_cce, type = "all", seed = 42)
#> Cross-sectional dependence tests
#> N = 15, T = 38
#>
#> statistic p.value
#> CD -2.676 0.007
#> CDw -0.714 0.475
#> CDw+ 2.247 0.025
#> CDstar 2.950 0.003
cd_test(fit_cce, type = "CDw+", reps = 30, seed = 42)
#> Cross-sectional dependence tests
#> N = 15, T = 38
#>
#> statistic p.value
#> CDw+ 4.490 0.000To test panel variables directly, select them explicitly through
.... Bare column names, quoted names, and character vectors
are supported. For a plain data frame, supply named id and
time column names. All test controls must also be named
because they follow ... in the data methods.
variable_tests <- cd_test(
dat, log_rgdpo, "log_hc", id = "id", time = "year"
)
variable_tests
#> Cross-sectional dependence tests by variable
#>
#> variable test statistic p.value N T
#> log_rgdpo CD 18.823 0.000 15 38
#> log_hc CD 52.602 0.000 15 38
variable_tests$log_rgdpo$tests$CD
#> $statistic
#> [1] 18.82336
#>
#> $p.value
#> [1] 4.860322e-79
#>
#> $N
#> [1] 15
#>
#> $T
#> [1] 38
#>
#> $pairs_used
#> [1] 105
vars <- c("log_rgdpo", "log_hc")
cd_test(dat, vars, id = "id", time = "year", type = "CDw",
reps = 30, seed = 42)
#> Cross-sectional dependence tests by variable
#>
#> variable test statistic p.value N T
#> log_rgdpo CDw 4.896 0.000 15 38
#> log_hc CDw 1.895 0.058 15 38Each variable is tested separately on its own available sample, with
no regression fitted. The result is always a named list of class
cd_test_list, even for one variable; printing it shows a
combined table with the sample dimensions for each variable. Seeded
tests apply the seed separately to each variable, making results
independent of selection order. A pdata.frame provides the
indexes automatically, including when its index columns have been
removed:
panel_dat <- plm::pdata.frame(dat, index = c("id", "year"), drop.index = TRUE)
cd_test(panel_dat, log_rgdpo, log_hc, type = "CDw+", reps = 30, seed = 42)Periods with no finite residual for any retained unit are outside the effective residual sample and are removed automatically. Partially observed periods are handled as follows:
- classical CD remains pairwise-complete under the default
na.action = "pairwise"; - CDw and CDw+ require a balanced matrix and otherwise error;
- CD* requires a balanced matrix and otherwise returns
NAwith a warning.
To evaluate all diagnostics on one common time sample, use:
cd_test(
fit_cce,
type = "all",
seed = 42,
na.action = "drop.incomplete.times"
)A rejection concerns the variable or residual dependence targeted by the selected test; it does not by itself identify the source of misspecification. A non-rejection does not prove residual independence, particularly in small samples or weak-power settings.
Practical limits
- Large- normal inference does not correct short- dynamic bias.
- CCE validity requires enough informative averages and suitable factor and loading conditions.
- Long-run ratios require a stable, nonzero AR denominator and an economically defensible long-run interpretation.
- Reported mean-group fit statistics summarize unit regressions; they are not pooled-regression goodness-of-fit measures.
- Saved fits from older releases should be refitted because corrected samples, transformations, and covariance calculations can change results.