The M-step puts its start inside the parameter box and clamps the optimum
it returns. NLopt refuses a start outside its bounds, and BOBYQA, which
works in coordinates scaled by the initial step, can return a point one
rounding error past a bound; given back as the next start, every M-step
from there failed and was reported as a failed E-step. A fit whose estimate
sat on such a bound lost iterations to the recovery and returned
loglik = NA, because the final E-step at the estimate failed the same way:
21 of 3,404 fits of the validation series' model-choice study (d, nd,
nk) ended so. On one of them (nd, 23 tips) every one of 591 failed calls
in two minutes started 5.6e-17 past a bound; with the start in the box the
same two minutes hold no failure and 58 conditioned iterations instead
of 27. Under the thinning proposal a final E-step that fails is also retried
with the attempt cap doubled, as a failed E-step inside the loop already
was, and the verbose messages of both give the error.
The M-step uses BOBYQA only under the exponential link (and a finite box);
under the linear link it uses subplex, as before. BOBYQA's quadratic model
stalls on the kinks of max(0, eta): on a 12-tip tree (dd, 30 draws) it
returned after 18 evaluations within 0.003 of its start, a point subplex
leaves at once for one 0.65 log-likelihood units higher after five EM
iterations on the same draws, whereas under the exponential link on the same
tree BOBYQA's point is the one subplex stays at. Found by the tests pinned
to linear-link fits after ef8a84e made BOBYQA the default.
estimate_rates(cond = "surface") on a clock gives its unconditioned start
at most control$surface_start_share (0.25) of max_time, so that the
conditioned continuation keeps the rest. In the validation series V6, 225 of
927 fits and in V7 123 of 992 ended on a 22-hour clock with no conditioned
iteration: the start, run to the stopping rule at up to 2400 draws, had
taken the clock (a median of 10-19 hours of it), and iterations, which
counts conditioned iterations only, read 0. The fit now records every stage
in conditioning$runs (seconds, iterations, draws, stop reason and trace
of the start; segments, surfaces and clades of each conditioned run), and
details$mcem is the trace of the whole run kept, every segment with its
segment number, not the last segment's alone.
estimate_rates(method = "mcem") without init_pars starts at the
constant-rate maximum-likelihood estimate of the tree (closed form,
conditioned on survival when cond is a function; .cr_mle()), with every
covariate coefficient at zero, instead of the middle of the parameter box.
On 40 constant-rate trees (80-150 tips, lambda = 0.5, mu = 0.2, conditioned
fits) the median absolute error in mu against the exact estimate falls from
0.089 to 0.002 and the median estimate of mu moves from 0.072 to 0.209
(exact 0.209); the fits take a median of 9 iterations.
New default stopping rule stop_rule = "projected": an iteration counts
toward patience when the relative step and the relative distance to the
fixed point projected from the last proj_window (8) iterates are both
below tol. The trace records the projected distance in the column
projected. "rel_change", the previous default, remains available.
The trees an E-step returns under the thinning proposal carry their lineage
ids and covariate columns again. The copy rcpp_mcem.cpp made of them had
only brts, n and t_ext, so under stop_rule = "mc_error" every batch
M-step of a D, ED or K model failed to build its covariate table, the
Monte Carlo error of the step was never computed (mc_z was NA on every
iteration), the draws never grew and no thinning fit of those models met
the rule: it ran to max_iter whatever the estimate did. The surrogate
proposal was not affected.
estimate_rates(cond = "surface") conditions a fit on survival for any
model and link: an unconditioned fit, then MCEM continued on local
quadratic surfaces of a direct Monte Carlo estimate of log P(S) around the
iterate, rebuilt whenever the iterate leaves the inner half of a surface's
box, with a second run from the nested start when the unconditioned
extinction intercept is on an edge of the box. The reported log-likelihood
subtracts a direct estimate of log P(S) at the estimate, and the fit's
conditioning records the surfaces, clades, seconds and the start kept.
It is the recommended conditioning for the lineage-level models, whose
survival GAM is not accurate enough (E43). The settings are control
entries named surface_*.
survival_mc() estimates log P(S) by direct simulation for any model, link
and sampling fraction, with common random numbers across parameter values;
survival_surface() fits the local surface, with predict() and print()
methods. predict_survival() and estimate_rates(cond = ) accept a
surface.
First release on CRAN. The package was rewritten after version 0.4
(2021--2024). The functions of that version (emphasis(), emphasis_de(),
sim_tree_pd_cpp(), augmentPD(), loglikelihood() and the rest) are gone;
the interface is the one described below.
estimate_rates(), simulate_tree(), auto_bounds()
and emphasis_pipeline() is now "exponential", under which the
intercepts are log rates; it was "linear". The model ladder, the
real-tree analyses, the exact comparison on real trees and the conditioned
estimability study use the exponential link; the earlier simulation studies
ran on the linear link and remain valid for it. Scripts relying on the linear default
must pass link = "linear", which stays supported.D likelihood on a bare branching-time vector counted one lineage too
many after an observed split, so a d model with zero D coefficients did
not reduce to cr; the alive lineages now follow the forward sweep by
lineage id. Trees given as phylo objects were not affected.simulate_tree(), replaces the separate forward simulators
and augmentation functions. It simulates trees forward, in batches over a
parameter matrix, or conditionally on an observed tree (augmentation).estimate_rates(), replaces emphasis() and
emphasis_de(), with the method as an argument.rho enters the
simulator, the likelihood and both augmentation samplers.cond argument of estimate_rates(): the
likelihood is divided by the survival probability of the clade, given as a
function of the parameters, a GAM or an nnet object. train_GAM() now
fits a binomial GAM of survival on the parameters; in version 0.4 it fitted
a GAM to simulated log-likelihoods. That GAM is not accurate enough to
condition on: in E43 (https://franciscorichter.github.io/emphasis/experiments/E43.html) its
90th-percentile error of log P(S) was 0.44 to 2.56 for every lineage-level
model at 2000, 5000 and 20 000 training clades, against 0.05 required.
Under cr and dd condition on the exact survival probability, passed as
a function (?estimate_rates gives the cr formula); under lineage-level
models on a direct Monte Carlo estimate, or not at all.D, evolutionary
distinctiveness ED (raw or centred) and the number of ancestral splits K
join diversity N. Version 0.4 read diversity and the clade's phylogenetic
diversity.compare_models(), lineage_table() and
covariate_path().num_threads is set.N, age-imbalance D, evolutionary
distinctiveness ED (raw or centred, EDc) and the number of ancestral
splits K. Models are named ("cr", "dd", "d", "nd", "ed",
"ned", "edc", "nedc", "k", "nk") or given as a formula such as
~ N + ED, on an exponential (the default) or linear link.estimate_rates() fits a model by Monte Carlo expectation-maximization
("mcem"), the cross-entropy method ("cem") or a GAM fitted to a
likelihood surface ("gam"). model = "all" fits the nested ladder of
models and ranks it.auto_bounds() builds a parameter box from the observed tree by forward
simulation.compare_models() compares fits by AIC and reports the Monte Carlo error of
each log-likelihood and the downward bias of a log of a Monte Carlo mean.lineage_table() returns the per-lineage design the likelihood is built on,
and covariate_path() traces a selection path over the covariates.cond takes is a function, a GAM from
train_GAM() or an nnet object, evaluated by predict_survival().diagnose_mcem(), diagnose_cem() and diagnose_gam() report convergence
and importance-sampling diagnostics.message() and can be
silenced with suppressMessages() or verbose = FALSE.emphasis_pipeline() runs on one thread unless control$num_threads is
set.simulate_tree()'s max_tries is the number of forward draws in all, the
first included: the default 1 is a single draw, unconditioned on
survival, and an extinct clade is redrawn until one survives or max_tries
draws have been made. max_tries = 0, which asked for a single draw when
the argument counted the draws after the first, is read as 1 with a
warning. auto_bounds() passes surv_tries (default 20) as max_tries,
and the survival GAM is trained on single draws.control$max_time binds inside the E-step and the M-step of an MCEM fit,
and the fit ends with stop_reason = "time_budget" and a clock note
naming the step it cut. Both samplers read the clock between draws and
inside a draw and stop drawing when it runs out; the iteration in progress
is discarded. The M-step's optimiser (NLopt's sbplx, under both samplers)
reads the same deadline before every evaluation of its objective and stops
once it has passed; the iteration then keeps the best point the optimiser
had evaluated when that point improves on the current iterate, and is
discarded otherwise. The E-step that evaluates the returned parameters gets
max_time again and, when it runs out, estimates the log-likelihood from
the draws it made, so a fit returns within about twice max_time.notes, the named notes estimate_rates() states about it.
When the surrogate proposal's mean-field iteration was relaxed below the
full step, ran under a damping other than "auto", or did not converge
within its sweep budget, the mean_field note gives the setting, the sweeps
taken, the relaxation and the final residual at the last E-step and whether
it converged, and print() shows it. Every E-step's mean field is in the
trace (mf_converged, mf_delta, mf_sweeps, mf_omega, mf_budget).control$damping defaults to "default", which is "auto" at complete
sampling (rho = 1) and a fixed relaxation of 1/8 when rho < 1. In
experiment E10c a fixed 1/8 converged 0 of 192 units within the 20 sweeps
at rho = 1 against 131 for "auto", at 1.4--3.3 times the E-step time
and no gain in effective sample, and below rho = 1 it converged more
(142 against 131 at rho = 0.2). An explicit "auto" or number is used as
given, and the mean_field note names the resolved setting and whether it
is the default.compare_models() chooses the model with the fewest parameters within
margin of the minimum AIC and, among those, the one with the lower AIC,
whatever order the fits are passed in.