> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/Metaculus/metaculus/llms.txt
> Use this file to discover all available pages before exploring further.

# Aggregation Methods

> Learn how Metaculus combines individual forecasts into community predictions

Metaculus aggregates individual forecasts to create the Community Prediction (CP), which often outperforms most individual forecasters. This guide explains the different aggregation methods and when each is used.

## Overview

Aggregation is the process of combining multiple individual forecasts into a single collective prediction. Metaculus uses sophisticated methods to weight and combine forecasts in ways that leverage the wisdom of crowds.

<Note>
  The community prediction is continuously updated as new forecasts are submitted or existing forecasts are updated.
</Note>

## Aggregation Methods

Metaculus supports four primary aggregation methods, defined in the `AggregationMethod` enum:

### Recency Weighted

**Method**: `recency_weighted`\
**Default**: Yes (for most questions)\
**Best for**: Standard questions with regular updates

<Tabs>
  <Tab title="How it works">
    Each forecast is weighted based on when it was made, with more recent forecasts receiving higher weight. This ensures the community prediction reflects the most current information and understanding.

    **Weighting formula:**

    * Recent forecasts get maximum weight
    * Older forecasts decay exponentially
    * The decay rate is calibrated to balance recency with stability
  </Tab>

  <Tab title="Advantages">
    * **Responsive**: Quickly adapts to new information
    * **Current**: Reflects latest expert consensus
    * **Incentivizes updates**: Encourages forecasters to maintain their predictions
    * **Reduces staleness**: Old, outdated forecasts have less influence
  </Tab>

  <Tab title="Disadvantages">
    * Can be volatile if few recent forecasts
    * May overweight recent overreactions
    * Less stable than unweighted methods
  </Tab>
</Tabs>

<Tip>
  Recency weighted aggregation is ideal for questions where information evolves over time, such as geopolitical events, technology developments, or market predictions.
</Tip>

### Unweighted

**Method**: `unweighted`\
**Best for**: Short-term questions, live forecasting, spot predictions

<Tabs>
  <Tab title="How it works">
    All active forecasts receive equal weight regardless of when they were made. The aggregate is a simple combination of all current predictions.

    **Process:**

    1. Collect all active forecasts
    2. Combine with equal weighting
    3. Generate aggregate distribution
  </Tab>

  <Tab title="Advantages">
    * **Stable**: Less sensitive to individual forecast changes
    * **Simple**: Easy to understand and compute
    * **Democratic**: Every forecaster's current view counts equally
    * **Robust**: Resistant to manipulation from single forecasters
  </Tab>

  <Tab title="Disadvantages">
    * May be slow to respond to new information
    * Old forecasts can drag down accuracy
    * Doesn't incentivize updating
  </Tab>
</Tabs>

<Note>
  Unweighted aggregation is recommended for questions that close within hours or days, or where forecasting happens in real-time during an event.
</Note>

### Single Aggregation

**Method**: `single_aggregation`\
**Best for**: Special aggregate-only questions, benchmarking

<Tabs>
  <Tab title="How it works">
    Instead of aggregating user forecasts, this method uses a single pre-computed aggregate forecast. This is typically used for:

    * Reference forecasts
    * Baseline predictions
    * Historical aggregates
    * External model predictions
  </Tab>

  <Tab title="Use cases">
    * **Benchmarking**: Compare community performance against a fixed baseline
    * **Historical data**: Preserve aggregate from a specific point in time
    * **Model forecasts**: Include algorithmic predictions
    * **Expert panels**: Single aggregate from a closed group
  </Tab>
</Tabs>

### Metaculus Prediction

**Method**: `metaculus_prediction`\
**Best for**: Official Metaculus forecast, enhanced aggregation

<Tabs>
  <Tab title="How it works">
    A proprietary Metaculus aggregation method that may incorporate:

    * Advanced weighting schemes
    * Historical forecaster performance
    * Calibration adjustments
    * Extremization
    * Other enhancements to standard aggregation

    The exact algorithm is periodically updated as Metaculus improves its aggregation techniques.
  </Tab>

  <Tab title="Advantages">
    * **Optimized**: Designed to maximize accuracy
    * **Sophisticated**: Uses multiple signals beyond recency
    * **Research-backed**: Based on forecasting research
    * **Performance-weighted**: Better forecasters may have more influence
  </Tab>
</Tabs>

<Warning>
  The Metaculus Prediction method is less transparent than other methods since the exact algorithm may evolve. For transparency and reproducibility, recency weighted or unweighted methods are preferred.
</Warning>

## Setting the Aggregation Method

Question authors set the default aggregation method using the `default_aggregation_method` field:

```python theme={null}
default_aggregation_method = models.CharField(
    max_length=20,
    choices=AggregationMethod.choices,
    default=AggregationMethod.RECENCY_WEIGHTED,
)
```

### Guidelines for Choosing

<AccordionGroup>
  <Accordion title="Use Recency Weighted when:">
    * Question is open for weeks or months
    * Information environment is dynamic
    * You want to encourage regular updates
    * This is the default for good reason - it works well for most questions
  </Accordion>

  <Accordion title="Use Unweighted when:">
    * Question closes within 24-72 hours
    * Forecasting happens live during an event
    * You want maximum stability
    * You're doing spot predictions
  </Accordion>

  <Accordion title="Use Single Aggregation when:">
    * You're providing a reference forecast
    * You want to preserve a historical aggregate
    * You're incorporating an external model
  </Accordion>

  <Accordion title="Use Metaculus Prediction when:">
    * You want Metaculus's best aggregate
    * Performance weighting is important
    * You trust Metaculus's aggregation research
  </Accordion>
</AccordionGroup>

## How Aggregation Works by Question Type

Different question types aggregate differently:

### Binary Questions

Aggregates the probability values:

```
CP = Aggregate(p₁, p₂, p₃, ..., pₙ)
```

Where each `pᵢ` is a forecaster's probability estimate.

<Tabs>
  <Tab title="Recency Weighted">
    ```
    CP = Σ(wᵢ × pᵢ) / Σ(wᵢ)
    ```

    Where `wᵢ` is the recency weight for forecast `i`.
  </Tab>

  <Tab title="Unweighted">
    ```
    CP = Σ(pᵢ) / n
    ```

    Simple mean of all active forecasts.
  </Tab>
</Tabs>

### Multiple Choice Questions

Aggregates probability distributions across options:

```
CP_option[j] = Aggregate(p₁[j], p₂[j], ..., pₙ[j])
```

Each option gets its own aggregated probability, and the final CP is normalized to sum to 100%.

<Note>
  When options change (added/removed), forecasts are automatically adjusted proportionally to maintain valid probability distributions.
</Note>

### Continuous Questions (Numeric, Date, Discrete)

Aggregates probability distributions (CDFs):

1. Each forecast is represented as a cumulative distribution function (CDF)
2. CDFs are combined using the chosen method
3. The result is a smooth aggregate CDF representing the community's collective belief

**For percentiles:**

* 25th percentile: 25% of the aggregate probability is below this value
* 50th percentile (median): The community's central estimate
* 75th percentile: 75% of the aggregate probability is below this value

## Aggregate Forecast Storage

Aggregates are stored in the `AggregateForecast` model:

```python theme={null}
class AggregateForecast(models.Model):
    question = models.ForeignKey(Question, ...)
    method = models.CharField(choices=AggregationMethod.choices)
    start_time = models.DateTimeField()  # When this aggregate became active
    end_time = models.DateTimeField()     # When it was superseded (null if current)
    forecast_values = ArrayField(...)     # The aggregate CDF/PMF
    forecaster_count = models.IntegerField()  # How many forecasters contributed
    # Additional statistics
    interval_lower_bounds = ArrayField(...)
    centers = ArrayField(...)
    interval_upper_bounds = ArrayField(...)
    means = ArrayField(...)
    histogram = ArrayField(...)
```

<Tip>
  Aggregate forecasts are versioned over time. You can query historical aggregates by filtering on `start_time` and `end_time`.
</Tip>

## Aggregate Statistics

Beyond the central aggregate, Metaculus computes additional statistics:

### Forecaster Count

The number of unique forecasters whose predictions contribute to the current aggregate.

### Percentile Ranges

For continuous questions:

* **25th-75th percentile range**: The middle 50% of the aggregate distribution
* **5th-95th percentile range**: The middle 90% of the aggregate distribution

### Histogram

A 100-bin histogram showing the shape of the aggregate distribution, useful for visualization.

## Community Prediction Visibility

Some questions hide the CP until a reveal time (`cp_reveal_time`):

```python theme={null}
@property
def is_cp_hidden(self):
    return (
        not self.resolution  # always show cp when resolved
        and self.cp_reveal_time
        and self.cp_reveal_time > timezone.now()
    )
```

This prevents anchoring and groupthink, encouraging independent thinking.

<Warning>
  When CP is hidden, you should form your own prediction before the reveal time. Research shows that seeing the CP first can bias your forecast.
</Warning>

## Bot Forecasts

Questions can include or exclude bot forecasts from aggregates:

```python theme={null}
include_bots_in_aggregates = models.BooleanField(default=False)
```

* **False** (default): Only human forecasts are aggregated
* **True**: Bot forecasts are included in the CP

<Note>
  Bot forecasts are marked with `is_bot=True` on the user account. Some bots are marked as `is_primary_bot=True` for special treatment.
</Note>

## Aggregation Performance

Research shows that aggregated forecasts typically:

* **Outperform** most individual forecasters
* **Are well-calibrated** when the crowd is diverse
* **Improve with more forecasters** (up to a point)
* **Benefit from forecaster diversity** in methods and information

### When Aggregation Works Best

<CardGroup cols={2}>
  <Card title="High Quality" icon="star">
    * Many active forecasters (50+)
    * Diverse viewpoints and methods
    * Regular updates
    * Clear resolution criteria
  </Card>

  <Card title="Conditions Matter" icon="triangle-exclamation">
    * Adequate information available
    * Forecasters have domain knowledge
    * No systematic biases
    * Sufficient time horizon
  </Card>
</CardGroup>

### When Aggregation Struggles

* Very few forecasters (less than 5)
* Highly correlated information sources
* Groupthink or information cascades
* Ambiguous resolution criteria
* Black swan events with no base rates

## Best Practices

<AccordionGroup>
  <Accordion title="For forecasters">
    1. **Forecast independently**: Form your view before checking the CP
    2. **Update regularly**: Keep your forecast current, especially with recency weighting
    3. **Diversify your approach**: Use different methods than others might
    4. **Explain your reasoning**: Help others understand alternative perspectives
  </Accordion>

  <Accordion title="For question authors">
    1. **Choose appropriate method**: Match the aggregation to the question timeline
    2. **Consider CP reveal time**: Hide CP initially for important questions
    3. **Exclude bots carefully**: Include only if bots add value
    4. **Monitor aggregate quality**: Check forecaster count and diversity
  </Accordion>

  <Accordion title="For tournament organizers">
    1. **Standardize methods**: Use consistent aggregation across similar questions
    2. **Document choices**: Explain why you chose each method
    3. **Test alternatives**: Compare methods on resolved questions
    4. **Consider scoring**: Some scoring methods work better with certain aggregations
  </Accordion>
</AccordionGroup>

## API Access

You can retrieve aggregate forecasts via the API:

```python theme={null}
# Get the latest aggregate for a question
agg = AggregateForecast.objects.filter(
    question=question,
    method=question.default_aggregation_method,
    end_time__isnull=True
).first()

# Get historical aggregates
aggs = AggregateForecast.objects.filter(
    question=question,
    method=question.default_aggregation_method
).order_by('start_time')
```

See the API reference for more details.

## Next Steps

<CardGroup cols={2}>
  <Card title="Making Predictions" icon="chart-line" href="/guides/making-predictions">
    Learn how to submit your own forecasts
  </Card>

  <Card title="Question Types" icon="list-check" href="/guides/question-types">
    Understand different prediction formats
  </Card>

  <Card title="Track Record" icon="trophy" href="/guides/track-record">
    See how aggregates are scored
  </Card>

  <Card title="API Reference" icon="code" href="/api/questions">
    Access aggregate data programmatically
  </Card>
</CardGroup>