broom 0.7.2 Unreleased

  • In broom 0.7.0, we introduced an error for objects that subclassed lm and relied on tidy.lm(). We’ve transitioned this error to a warning.
  • Various bug fixes and improvements to documentation and errors.

broom 0.7.1 2020-10-01

While broom 0.7.1 is a minor release, it includes a number of exciting new features and bug fixes!

New tidiers

  • Add tidiers for margins objects. (#700 by @grantmcdermott)
  • Added tidier methods for mlogit objects (#887 by @gregmacfarlane)
  • Add glance.coeftest() method (#932 by @grantmcdermott)

Improvements to existing tidiers

One of the more major improvements in this release is the addition of the interval argument to some augment methods for confidence, prediction, and credible intervals. These columns will be consistently labeled .lower and .upper! (#908 by @grantmcdermott, #925 by @bwiernik)

In addition…

  • Extended the glance.aov() method to include an r.squared column!
  • glance.survfit() now passes ... to summary.survfit() to allow for adjustment of RMST and other measures (#880 by @vincentarelbundock)
  • Several unsupported model objects that subclass glm and lm now error more informatively.
  • A number of improvements to documentation throughout the package.

Bug fixes

  • Fixed newdata warning message in augment.*() output when the newdata didn’t contain the response variable—augment methods no longer expect the response variable in the supplied newdata argument. (#897 by @rudeboybert)
  • Fixed a bug related to tidy.geeglm() not being sensitive to the exponentiate argument (#867)
  • Fixed augment.fixest() returning residuals in the .fitted column. The method also now takes a type.residuals argument and defaults to the same type.predict argument as the fixest predict() method. (#877 by @karldw)
  • Fix tidy.felm confidence interval bug. Replaces “robust” argument with “se.type”. (#919 by @grantmcdermott; supersedes #818 by @kuriwaki)
  • Fix a bug in tidy.drc() where some term labels would result in the overwriting of entries in the curve column (#914)
  • Fixed bug related to univariate zoo series in tidy.zoo() (#916 by @WillemVervoort)
  • Fixed a bug related to tidy.prcomp() assigning the wrong PC labels from “loadings” and “scores” matrices (#910 by @tavareshugo)
  • Fixed tidy.polr() bug where p-values could only be returned if exponentiate = FALSE.

Deprecations

We followed through with the planned deprecation of character vector tidiers in this release. Other vector tidiers that were soft-deprecated in 0.7.0 will be fully deprecated in a later release.

broom 0.7.0 2020-07-09

broom 0.7.0 is a major release with a large number of new tidiers, soft-deprecations, and planned hard-deprecations of functions and arguments.

Big picture changes

  • We have changed how we report degrees of freedom for lm objects (#212, #273). This is especially important for instructors in statistics courses. Previously the df column in glance.lm() reported the rank of the design matrix. Now it reports degrees of freedom of the numerator for the overall F-statistic. This is equal to the rank of the model matrix minus one (unless you omit an intercept column), so the new df should be the old df minus one.

  • We are moving away from supporting summary.*() objects. In particular, we have removed tidy.summary.lm() as part of a major overhaul of internals. Instead of calling tidy() on summary-like objects, please call tidy() directly on model objects moving forward.

  • We have removed all support for the quick argument in tidy() methods. This is to simplify internals and is for maintainability purposes. We anticipate this will not influence many users as few people seemed to use it. If this majorly cramps your style, let us know, as we are considering a new verb to return only model parameters. In the meantime, stats::coef() together with tibble::enframe() provides most of the functionality of tidy(..., quick = TRUE).

  • All conf.int arguments now default to FALSE, and all conf.level arguments now default to 0.95. This should primarily affect tidy.survreg(), which previously always returned confidence intervals, although there are some others.

  • Tidiers for emmeans-objects use the arguments conf.int and conf.level instead of relying on the argument names native to the emmeans::summary()-methods (i.e., infer and level). Similarly, multcomp-tidiers now include a call to summary() as previous behavior was akin to setting the now removed argument quick = TRUE. Both families of tidiers now use the adj.p.value column name when appropriate. Finally, emmeans-, multcomp-, and TukeyHSD-tidiers now consistently use the column names contrast and null.value instead of comparison, level1 and level2, or lhs and rhs (see #692).

Deprecations

This release of broom soft-deprecates the following functions and tidier methods:

We have also gone forward with our planned mixed model deprecations, and have removed the following methods, which now live in broom.mixed:

  • tidy.brmsfit()
  • tidy.merMod(), glance.merMod(), augment.merMod()
  • tidy.lme(), glance.lme(), augment.lme()
  • tidy.stanreg(), glance.stanreg()
  • tidyMCMC(), tidy.rjags(), tidy.stanfit()

Minor breaking changes

  • augment.factanal() now returns a tibble with columns names .fs1, .fs2, …, instead of factor1, factor2, … (#650)

  • We have renamed the output of augment.htest(). In particular, we have renamed the .residuals column to .resid and the .stdres to .std.resid for consistency. These changes will only affect chi-squared tests.

  • tidy.ridgelm() now always return a GCV column and never returns an xm column. (#533 by @jmuhlenkamp)

  • tidy.dist() no longer supports the upper argument.

A refactoring of augment() methods

The internals of augment.*() methods have largely been overhauled.

  • If you pass a dataset to augment() via the data or newdata arguments, you are now guaranteed that the augmented dataset will have exactly the same number of rows as the original dataset. This differs from previous behavior primarily when there are missing values. Previously augment() would drop rows containing NA. This should no longer be the case.

  • augment.*() methods no longer accept an na.action argument.

  • In previous versions, several augment.*() methods inherited the augment.lm() method, but required additions to the augment.lm() method itself. We have shifted away from this approach in favor of re-implementing many augment.*() methods as standalone methods making use of internal helper functions. As a result, augment.lm() and some related methods have deprecated (previously unused) arguments.

  • augment() tries to give an informative error when data isn’t the original training data.

  • The .resid column in the output of augment().* methods is now consistently defined as y - y_hat

New tidiers

  • anova objects from the car package (#754)
  • pam objects from the cluster package (#637 by @abbylsmith)
  • drm objects from the drc package (#574 by @edild)
  • summary_emm objects from the emmeans package (#691 by @crsh)
  • epi.2by2 objects from the epiR package (#711)
  • fixest objects from the fixest package (#785 by @karldw)
  • regsubsets objects from the leaps package (#535)
  • lm.beta objects from the lm.beta package (#545 by @mattle24)
  • rma objects from the metafor package (#674 by @malcolmbarrett, @softloud)
  • mfx, logitmfx, negbinmfx, poissonmfx, probitmfx, and betamfx objects from themfx package (#700 by @grantmcdermott)
  • lmrob and glmrob objects from the robustbase package (#205, #505)
  • sarlm objects from the spatialreg package (#847 by @gregmacfarlane and @petrhrobar)
  • speedglm objects from the speedglm package (#685)
  • svyglm objects from the survey package (#611)
  • systemfit objects from the systemfit package (by @jaspercooper)
  • We have restored a simplified version of glance.aov(), which used to inherit from the glance.lm() method and now contains only the following columns: logLik, AIC, BIC, deviance, df.residual, and nobs (see #212). Note that tidy.aov() gives more complete information about degrees of freedom in an aov object.

Improvements to existing tidiers

Bug fixes

  • Bug fix for tidy.survreg() when robust is set to TRUE in model fitting (#842, #728)
  • Bug fixes in glance.lavaan(): address confidence interval error (#577) and correct reported nobs and norig (#835)
  • Bug fix in muhaz tidiers to ensure output is always a tibble (#824)
  • Several glance.*() methods have been refactored in order to return a one-row tibble even when the model matrix is rank-deficient (#823)
  • Bug fix to return confidence intervals correct in tidy.drc() (#798)
  • Added default methods for objects that subclass glm and lm in order to error more informatively. (#749, #736, #708, #186)
  • Bug fix to allow augment.kmeans() to work with masked data (#609)
  • Bug fix to allow augment.Mclust() to work on univariate data (#490)
  • Bug fix to allow tidy.htest() to supports equal variances (#608)
  • Bug fix to better allow tidy.boot() to support confidence intervals (#581)
  • Bug fix for tidy.polr() when passed conf.int = TRUE (#498)

Other changes

  • Many glance() methods now return a nobs column, which contains the number of data points used to fit the model! (#597 by @vincentarelbundock)

  • tidy() no longer checks for a log or logit link when exponentiate = TRUE, and we have refactored to remove extraneous exponentiate arguments. If you set exponentiate = TRUE, we assume you know what you are doing and that you want exponentiated coefficients (and confidence intervals if conf.int = TRUE) regardless of link function.

  • We now use rlang::arg_match() when possible instead of arg.match() to give more informative errors on argument mismatches.

  • The package’s site has moved from https://broom.tidyverse.org/ to https://broom.tidymodels.org/.

  • Revised several vignettes and moved them to the tidymodels.org website. The existing vignettes will now simply link to the revised versions.

  • Many improvements to consistency and clarity of documentation.

  • Various warnings resulting from changes to the tidyr API in v1.0.0 have been fixed. (#870)

  • Removed dependencies on reshape2 and superseded functions in dplyr.

  • All documentation now links to help files rather than topics.

For developers and contributors

  • Moved core tests to the modeltests package.

  • Generally, after this release, the broom dev team will first ask that attempts to add tidier methods supporting a model object are first directed to the model-owning package. An article describing best practices in doing so can be found on the {tidymodels} website at https://www.tidymodels.org/learn/develop/broom/, and we will continue adding additional resources to that article as we develop them. In the case that the maintainer is uninterested in taking on the tidier methods, please note this in your issue or PR.

  • Added a new vignette discussing how to implement new tidier methods in non-broom packages.

broom 0.5.6 2020-04-20

  • Fix failing CRAN checks to due tibble 3.0.0 release. Removed xergm dependency.

broom 0.5.5 2020-02-29

  • Remove tidiers for robust package and drop robust dependency (temporarily)

broom 0.5.4 2020-01-27

  • Fixes failing CRAN checks as the joineRML package has been removed from CRAN

broom 0.5.3 2019-12-14

  • Fixes failing CRAN checks due to new matrix classing in R 4.0.0

broom 0.5.2 2019-04-07

  • Fixes failing CRAN checks

  • Changes to accommodate ergm 3.10 release. tidy.ergm() no longer has a quick argument. The old default of quick = FALSE is now the only option.

broom 0.5.1 2018-12-05

broom 0.5.0 2018-07-17

Tidiers now return tibble::tibble()s. This release also includes several new tidiers, new vignettes and a large number of bug fixes. We’ve also begun to more rigorously define tidier specifications: we’ve laid part of the groundwork for stricter and more consistent tidying, but the new tidier specifications are not yet complete. These will appear in the next release.

Additionally, users should note that we are in the process of migrating tidying methods for mixed models and Bayesian models to broom.mixed. broom.mixed is not on CRAN yet, but all mixed model and Bayesian tidiers will be deprecated once broom.mixed is on CRAN. No further development of mixed model tidiers will take place in broom.

Breaking changes

Almost all tidiers should now return tibbles rather than data.frames. Deprecated tidying methods, Bayesian and mixed model tidiers still return data.frames.

Users are mostly to experience issues when using augment in situations where tibbles are stricter than data frames. For example, specifying model covariates as a matrix object will now error:

library(broom)
library(quantreg)

fit <- rq(stack.loss ~ stack.x, tau = .5)
broom::augment(fit)
#> Error: Column `stack.x` must be a 1d atomic vector or a list

This is because the default data argument data = model.frame(fit) cannot be coerced to tibble.

Another consequence of this is that augment.survreg and augment.coxph from the survival package now require that the user explicitly passes data to either the data or newdata arguments.

These restrictions will be relaxed in an upcoming release of broom pending support for matrix-columns in tibbles.

Developers are likely to experience issues:

  • subsetting tibbles with [, which returns a tibble rather than a vector.

  • setting rownames on tibbles, which is deprecated.

  • using matrix and vector tidiers, now deprecated.

  • handling the additional tibble classes tbl_df and tbl beyond the data.frame class

  • linking to defunct documentation files – broom recently moved all tidiers to a roxygen2 template based documentation system.

New vignettes

This version of broom includes several new vignettes:

Several old vignettes have also been updated:

Deprecations

  • Matrix and vector tidiers have been deprecated in favor of tibble::as_tibble and tibble::enframe

  • Dataframe tidiers and rowwise dataframe tidiers have been deprecated

  • bootstrap() has been deprecated in favor of the rsample

  • inflate has been removed from broom

Other changes

Contributors

Many many thanks to all the following for their thoughtful comments on design, bug reports and PRs! The community of broom contributors has been kind, supportive and insightful and I look forward to working you all again!

@atyre2, @batpigandme, @bfgray3, @bmannakee, @briatte, @cawoodjm, @cimentadaj, @dan87134, @dgrtwo, @dmenne, @ekatko1, @ellessenne, @erleholgersen, @ethchr, @Hong-Revo, @huftis, @IndrajeetPatil, @jacob-long, @jarvisc1, @jenzopr, @jgabry, @jimhester, @josue-rodriguez, @karldw, @kfeilich, @larmarange, @lboller, @mariusbarth, @michaelweylandt, @mine-cetinkaya-rundel, @mkuehn10, @mvevans89, @nutterb, @ShreyasSingh, @stephlocke, @strengejacke, @topepo, @willbowditch, @WillemSleegers, @wilsonfreitas, and @MatthieuStigler

broom 0.4.4 2018-03-29

  • Fixed gam tidiers to work with “Gam” objects, due to an update in gam 1.15. This fixes failing CRAN tests

  • Improved test coverage (thanks to #267 from Derek Chiu)

broom 0.4.3 2017-11-20

  • Changed the deprecated dplyr::failwith to purrr::possibly

  • augment and glance on NULLs now return an empty data frame

  • Deprecated the inflate() function in favor of tidyr::crossing

  • Fixed confidence intervals in the gmm tidier (thanks to #242 from David Hugh-Jones)

  • Fixed a bug in bootstrap tidiers (thanks to #167 from Jeremy Biesanz)

  • Fixed tidy.lm with quick = TRUE to return terms as character rather than factor (thanks to #191 from Matteo Sostero)

  • Added tidiers for ivreg objects from the AER package (thanks to #245 from David Hugh-Jones)

  • Added tidiers for survdiff objects from the survival package (thanks to #147 from Michał Bojanowski)

  • Added tidiers for emmeans from the emmeans package (thanks to #252 from Matthew Kay)

  • Added tidiers for speedlm and speedglm from the speedglm package (#685, thanks to #248 from David Hugh-Jones)

  • Added tidiers for muhaz objects from the muhaz package (thanks to #251 from Andreas Bender)

  • Added tidiers for decompose and stl objects from stats (thanks to #165 from Aaron Jacobs)

broom 0.4.2 2017-02-13

  • Added tidiers for lsmobj and ref.grid objects from the lsmeans package

  • Added tidiers for betareg objects from the betareg package

  • Added tidiers for lmRob and glmRob objects from the robust package

  • Added tidiers for brms objects from the brms package (thanks to #149 from Paul Buerkner)

  • Fixed tidiers for orcutt 2.0

  • Changed tidy.glmnet to filter out rows where estimate == 0.

  • Updates to rstanarm tidiers (thanks to #177 from Jonah Gabry)

  • Fixed issue with survival package 2.40-1 (thanks to #180 from Marcus Walz)

broom 0.4.1 2016-06-24

  • Added AppVeyor, codecov.io, and code of conduct

  • Changed name of “NA’s” column in summaryDefault output to “na”

  • Fixed tidy.TukeyHSD to include term column. Also added separate.levels argument, with option to separate comparison into level1 and level2

  • Fixed tidy.manova to use correct column name for test (previously, always pillai)

  • Added kde_tidiers to tidy kernel density estimates

  • Added orcutt_tidiers to tidy the results of cochrane.orcutt orcutt package

  • Added tidy.dist to tidy the distance matrix output of dist from the stats package

  • Added tidy and glance for lmodel2 objects from the lmodel2 package

  • Added tidiers for poLCA objects from the poLCA package

  • Added tidiers for sparse matrices from the Matrix package

  • Added tidiers for prcomp objects

  • Added tidiers for Mclust objects from the Mclust package

  • Added tidiers for acf objects

  • Fixed to be compatible with dplyr 0.5, which is being submitted to CRAN

broom 0.4.0 2015-11-30

  • Added tidiers for geeglm, nlrq, roc, boot, bgterm, kappa, binWidth, binDesign, rcorr, stanfit, rjags, gamlss, and mle2 objects.

  • Added tidy methods for lists, including u, d, v lists from svd, and x, y, z lists used by image and persp

  • Added quick argument to tidy.lm, tidy.nls, and tidy.biglm, to create a smaller and faster version of the output.

  • Changed rowwise_df_tidiers to allow the original data to be saved as a list column, then provided as a column name to augment. This required removing data from the augment S3 signature. Also added tests-rowwise.R

  • Fixed various issues in ANOVA output

  • Fixed various issues in lme4 output

  • Fixed issues in tests caused by dev version of ggplot2

broom 0.3.7 2015-05-06

  • Added tidiers for “plm” (panel linear model) objects from the plm package.

  • Added tidy.coeftest for coeftest objects from the lmtest package.

  • Set up tidy.lm to work with “mlm” (multiple linear model) objects (those with multiple response columns).

  • Added tidy and glance for “biglm” and “bigglm” objects from the biglm package.

  • Fixed bug in tidy.coxph when one-row matrices are returned

  • Added tidy.power.htest

  • Added tidy and glance for summaryDefault objects

  • Added tidiers for “lme” (linear mixed effects models) from the nlme package

  • Added tidy and glance for multinom objects from the nnet package.

broom 0.3.6 2015-02-18

  • Fixed bug in tidy.pairwise.htest, which now can handle cases where the grouping variable is numeric.

  • Added tidy.aovlist method. This added stringr package to IMPORTS to trim whitespace from the beginning and end of the term and stratum columns. This also required adjusting tidy.aov so that it could handle strata that are missing p-values.

  • Set up glance.lm to work with aov objects along with lm objects.

  • Added tidy and glance for matrix objects, with tidy.matrix converting a matrix to a data frame with rownames included, and glance.matrix returning the same result as glance.data.frame.

  • Changed DESCRIPTION to new format

broom 0.3.5 2015-01-05

  • Fixed small bug in felm where the .fitted and .resid columns were matrices rather than vectors.

  • Added tidiers for rlm (robust linear model) and gam (generalized additive model) objects, including adjustments to “lm” tidiers in order to handle them. See ?rlm_tidiers and ?gam_tidiers for more.

  • Removed rownames from tidy.cv.glmnet output

broom 0.3.4 2014-11-23

  • The behavior of augment, particularly with regard to missing data and the na.exclude argument, has through the use of the augment_columns function been made consistent across the following models:

    • lm

    • glm

    • nls

    • merMod (lme4)

    • survreg (survival)

    • coxph (survival)

Unit tests in tests/testthat/test-augment.R were added to ensure consistency across these models.

  • tidy, augment and glance methods were added for rowwise_df objects, and are set up to apply across their rows. This allows for simple patterns such as:

regressions <- mtcars %>% group_by(cyl) %>% do(mod = lm(mpg ~ wt, .)) regressions %>% tidy(mod) regressions %>% augment(mod)

See ?rowwise_df_tidiers for more.

  • Added tidy and glance methods for Arima objects, and tidy for pairwise.htest objects.

  • Fixes for CRAN: change package description to title case, removed NOTES, mostly by adding globals.R to declare global variables.

  • This is the original version published on CRAN.

broom 0.3 Unreleased

  • Tidiers have been added for S3 objects from the following packages:

    • lme4

    • glmnet

    • survival

    • zoo

    • felm

    • MASS (ridgelm objects)

  • tidy and glance methods for data.frames have also been added, and augment.data.frame produces an error (rather than returning the same data.frame).

  • stderror has been changed to std.error (affects many functions) to be consistent with broom’s naming conventions for columns.

  • A function bootstrap has been added based on this example, to perform the common use case of bootstrapping models.

broom 0.2 Unreleased

  • Added “augment” S3 generic and various implementations. “augment” does something different from tidy: it adds columns to the original dataset, including predictions, residuals, or cluster assignments. This was originally described as “fortify” in ggplot2.

  • Added “glance” S3 generic and various implementations. “glance” produces a one-row data frame summary, which is necessary for tidy outputs with values like R^2 or F-statistics.

  • Re-wrote intro broom vignette/README to introduce all three methods.

  • Wrote a new kmeans vignette.

  • Added tidying methods for multcomp, sp, and map objects (from fortify-multcomp, fortify-sp, and fortify-map from ggplot2).

  • Because this integrates substantial amounts of ggplot2 code (with permission), added Hadley Wickham as an author in DESCRIPTION.