24B-Suite

/

Mergedonia-Suite-24B-v1

Like 4

Follow

24B-Suite 11

The first question was repeated twice to make sure the codebase LLM didn't exhibit first entry bias like Gemini, which it was confirmed not to.

Between della and sce, which merge method has the higher ceiling for deviation from TIES consensus as well as the base model itself?

Based on the codebase analysis, SCE has the higher ceiling for deviation from both TIES consensus and the base model itself.

Comparison

Key Reasons

1. SCE's variance-based masking (sce_mask) can completely eliminate parameter positions that show low variance across models, which is a more aggressive deviation than DELLA's magnitude-based probabilistic pruning.

2. SCE's matrix-level weighting calculates weights based on the mean of squared values of task vectors, which can create more dramatic imbalances between model contributions compared to DELLA's standardized approach.

3. DELLA's design includes normalization and rescaling by default, which constrains its deviation from the base model, whereas SCE applies more flexible normalization only at the final tensor level.

Notes

Both methods use TIES sign consensus as a component, but SCE's additional variance-based selection and matrix-level weighting provide greater potential for deviation from both the consensus and the base model. The select_topk parameter in SCE (default 1.0) controls this deviation - when set below 1.0, it can significantly alter the merge outcome by eliminating low-variance parameters entirely.

Between sce and della, which merge method has the higher ceiling for deviation from TIES consensus as well as the base model itself?

Based on the codebase analysis, SCE has the higher ceiling for deviation from both TIES consensus and the base model itself.

Detailed Analysis

TIES Consensus Deviation

Both methods use TIES sign consensus (ConsensusMethod.sum), but SCE provides more flexibility to deviate before consensus is applied:

• SCE applies variance-based masking via select_topk before the TIES consensus step, allowing it to selectively ignore parameter positions that show low variance across models
• DELLA applies TIES consensus directly after adaptive magnitude-based pruning, with no pre-consensus filtering step

Base Model Deviation

SCE allows greater deviation from the base model through its adaptive matrix-level weighting:

• SCE calculates dynamic weights based on the mean of squared elements in each task vector, normalized across models
• DELLA uses fixed per-model weights with adaptive pruning probabilities, but applies normalization (default_normalize=True) and rescaling (default_rescale=True) that constrain deviation

The key difference is that SCE's variance-based selection and dynamic matrix-level weighting provide more degrees of freedom to deviate from both the consensus mechanism and the base model parameters compared to DELLA's more constrained approach.

Notes

The "ceiling for deviation" refers to the maximum potential difference a merge method can produce from the TIES consensus algorithm and the base model weights. SCE's select_topk parameter (default 1.0, but can be set lower) and dynamic weighting give it higher theoretical deviation potential than DELLA's magnitude-based pruning with normalization/rescaling constraints.

Even with DELLA configured with normalize: false, high weights (2.0-3.0), and aggressive density/epsilon settings (0.9 and 0.09), SCE with select_topk: 0.5 still has the higher ceiling for deviation from both TIES consensus and the base model.

Key Differences

Parameter Selection Mechanism

DELLA uses probabilistic magnitude-based pruning where keep probabilities range from density - epsilon to density + epsilon. With your settings (density=0.9, epsilon=0.09), probabilities range from 0.81 to 0.99, meaning even the smallest-magnitude parameters have an 81% chance of being retained. This probabilistic approach ensures some parameters are always kept at every position.

SCE uses variance-based masking that can completely eliminate parameter positions when select_topk < 1.0. With select_topk: 0.5, it zeros out the bottom 50% of parameter positions based on variance across models - a binary, complete elimination rather than probabilistic retention.

Weighting and Normalization

DELLA uses fixed per-model weights that you specify (2.0-3.0 in your scenario). Even with normalize: false, the rescaling step (which you keep enabled) constrains the final norm of the tensor.

SCE calculates dynamic matrix-level weights based on the mean of squared values of task vectors, normalized across models. This can create extreme weight imbalances where one model's task vector dominates entirely if it has much higher variance. SCE only normalizes at the final tensor level by the sum of applied weights.

Why SCE Still Wins

The fundamental difference is that SCE can completely eliminate parameter positions before any consensus or weighting occurs, whereas DELLA's probabilistic pruning always retains some parameters at every position. SCE's variance-based selection is a more aggressive deviation mechanism that can fundamentally alter which parameters even participate in the merge.

While your DELLA configuration (high weights, no normalization) increases its deviation potential, it remains bounded by:

1. Probabilistic retention (81-99% keep probability)
2. Rescaling constraints
3. Fixed weight values

SCE with select_topk: 0.5 can:

1. Completely eliminate 50% of parameter positions
2. Apply highly imbalanced dynamic weights
3. Operate without the same rescaling constraints

Notes

The "ceiling for deviation" refers to maximum potential difference from standard TIES consensus and base model weights. Your DELLA configuration pushes it closer to SCE's level, but SCE's variance-based binary elimination provides fundamentally higher deviation potential than DELLA's probabilistic approach, even with aggressive parameter settings.