> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/Metaculus/metaculus/llms.txt
> Use this file to discover all available pages before exploring further.

# Bot Forecasting

> Understanding automated forecasting bots and their role in aggregates

## Overview

Metaculus supports automated forecasting bots - algorithmic forecasters that can submit predictions programmatically. Bots play an important role in the platform's forecasting ecosystem and benchmarking.

## What are Forecasting Bots?

Bots are User accounts with `is_bot=True` that:

* Submit forecasts programmatically via API
* Run automated forecasting algorithms
* Don't require human intervention
* Can be owned by individual users or Metaculus

**Common Bot Types:**

* **Baseline models**: Simple statistical forecasts
* **Machine learning models**: Trained on historical data
* **News-driven models**: React to information streams
* **Ensemble models**: Combine multiple approaches
* **Research bots**: Experimental forecasting algorithms

## Bot Ownership Model

### Personal Bots

Users can create and manage their own bots:

<Accordion title="Creating Your Bot">
  **Requirements:**

  * Active Metaculus account
  * API access token
  * Forecasting algorithm implementation

  **Limits:**

  * Standard users: Up to 5 bots per account
  * First bot automatically marked as "primary bot"
  * Superusers: Unlimited bots

  **Creating a Bot:**

  ```python theme={null}
  from users.services.bots_management import create_bot

  bot = create_bot(
      bot_owner=user,
      username="my_forecasting_bot",
      # Optional: additional user fields
  )
  ```

  **Bot Properties:**

  * `is_bot = True`
  * `bot_owner` = Your user account
  * `is_primary_bot` = True for first bot, False for others
  * Inherits language and theme preferences from owner
</Accordion>

### Primary vs Non-Primary Bots

**Primary Bot:**

* Your main bot account
* Automatically set for your first bot
* Included in some aggregations
* Full leaderboard participation

**Non-Primary Bots:**

* Secondary/experimental bots
* May be excluded from certain leaderboards
* Useful for testing algorithms
* Can be promoted to primary if needed

## Bot Integration in Aggregates

### The `include_bots_in_aggregates` Flag

Each question has an `include_bots_in_aggregates` boolean field that controls whether bot forecasts are included in the community aggregate.

<Accordion title="How the Flag Works">
  **When `include_bots_in_aggregates = True`:**

  * Bot forecasts included in aggregation calculations
  * Geometric mean computed over humans + bots
  * Affects Recency Weighted, Unweighted, etc.
  * Bot predictions visible in community forecast

  **When `include_bots_in_aggregates = False` (default):**

  * Bot forecasts excluded from aggregation
  * Only human forecasters contribute
  * Bots can still forecast (for scoring/research)
  * Bot predictions tracked separately

  **Setting the Flag:**

  Question authors can enable bot inclusion:

  ```json theme={null}
  {
    "include_bots_in_aggregates": true
  }
  ```

  **Use Cases for Enabling:**

  * Bot benchmarking questions
  * Hybrid human-AI forecasting
  * Testing AI contribution to wisdom of crowds
  * Questions where bot perspective adds value
</Accordion>

### Impact on Scoring

Bot inclusion affects score calculations:

**Peer Score:**

* If bots included: Your forecast compared to human+bot geometric mean
* If bots excluded: Your forecast compared to human-only geometric mean
* Bots themselves always scored against appropriate baseline

**Baseline Score:**

* Unaffected by `include_bots_in_aggregates`
* Always scored against statistical baseline
* Same calculation for humans and bots

**Example:**

```python theme={null}
# In score_math.py
base_forecasts = user_forecasts.exclude_non_primary_bots()

if not question.include_bots_in_aggregates:
    base_forecasts = base_forecasts.exclude(author__is_bot=True)

geometric_means = get_geometric_means(base_forecasts)
```

## Bot Leaderboards

Metaculus maintains separate leaderboards for bots:

### Global Bot Leaderboard

**Purpose:**

* Rank bot forecasting accuracy
* Compare bot algorithms
* Benchmark machine vs human performance

**Score Type:**

* Uses Baseline Score (not Peer Score)
* Avoids bots gaming each other
* Fair comparison across all questions

**Eligibility:**

* Must be marked as bot (`is_bot=True`)
* Primary bots included by default
* Non-primary bots may be excluded

**Updates:**
Run via management command:

```bash theme={null}
python manage.py update_global_bot_leaderboard
```

### Tournament Bot Status

Tournaments can configure bot participation:

<Accordion title="Bot Leaderboard Status Options">
  **Project-Level Setting:**

  ```python theme={null}
  class BotLeaderboardStatus(models.TextChoices):
      EXCLUDE_ALL = "exclude_all"  # No bots on leaderboard
      INCLUDE_PRIMARY = "include_primary"  # Only primary bots
      INCLUDE_ALL = "include_all"  # All bots
  ```

  **Inheritance:**

  1. Leaderboard's `bot_status` field (if set)
  2. Project's `bot_leaderboard_status` (if leaderboard field not set)
  3. Default exclusion policy

  **Use Cases:**

  * `EXCLUDE_ALL`: Human-only competition
  * `INCLUDE_PRIMARY`: Allow main bot entries
  * `INCLUDE_ALL`: Open to all bots, research competitions
</Accordion>

## Viewing Bot Forecasts

### On Question Pages

When `include_bots_in_aggregates = True`:

* Bot forecasts appear in aggregate chart
* Forecaster count includes bots
* "Includes bots" indicator shown
* Can toggle bot visibility in some views

### In Aggregation Explorer

**Bot Toggle:**

Some aggregation methods support bot filtering:

1. Select aggregation method (e.g., Recency Weighted)
2. Toggle **Include Bots** checkbox
3. Compare human-only vs human+bot aggregates
4. Useful for understanding bot impact

**Availability:**

* Recency Weighted: ✓ Supports bot toggle
* Unweighted: ✓ Supports bot toggle
* Single Aggregation: ✓ Supports bot toggle
* Metaculus Pros: ✗ No bot toggle (human-only by definition)
* Medalists: ✗ No bot toggle (medals are human-only)

### Download Question Data

Export options for bot forecasts:

```json theme={null}
{
  "include_bots": "default",  // Use question's setting
  "include_bots": "true",     // Force include bots
  "include_bots": "false"     // Force exclude bots
}
```

CSV exports include `is_bot` column to identify bot forecasts.

## Creating a Forecasting Bot

### Step 1: Set Up Bot Account

**Via Settings Page:**

1. Go to **Account Settings**
2. Navigate to **Bots** section
3. Click **Create Bot**
4. Enter bot username
5. Save API credentials securely

**Via API:**

Use the bot management service:

```python theme={null}
bot = create_bot(
    bot_owner=request.user,
    username="my_algorithm_v1"
)

# Get API token
token = Token.objects.create(user=bot)
```

### Step 2: Implement Forecasting Logic

**Basic Structure:**

```python theme={null}
import requests

BOT_TOKEN = "your_bot_api_token"
BASE_URL = "https://www.metaculus.com/api"

def make_forecast(question_id, probability):
    """Submit a forecast for a binary question."""
    headers = {"Authorization": f"Token {BOT_TOKEN}"}
    data = {
        "probability_yes": probability,
    }
    response = requests.post(
        f"{BASE_URL}/questions/{question_id}/forecast/",
        headers=headers,
        json=data
    )
    return response.json()

def get_open_questions():
    """Fetch open questions to forecast on."""
    headers = {"Authorization": f"Token {BOT_TOKEN}"}
    response = requests.get(
        f"{BASE_URL}/questions/",
        headers=headers,
        params={"status": "open"}
    )
    return response.json()

def run_bot():
    """Main bot loop."""
    questions = get_open_questions()
    for question in questions:
        # Your forecasting algorithm here
        probability = my_forecasting_algorithm(question)
        make_forecast(question['id'], probability)
```

### Step 3: Deploy and Monitor

**Deployment Options:**

* Serverless functions (AWS Lambda, Google Cloud Functions)
* Scheduled scripts (cron jobs)
* Continuous services (Docker containers)
* GitHub Actions

**Monitoring:**

* Track bot forecast volume
* Monitor error rates
* Check scoring performance
* Review calibration regularly

## Bot Best Practices

<Tip>
  **Building Effective Bots:**

  1. **Start simple**: Begin with baseline models
  2. **Test thoroughly**: Use resolved questions for backtesting
  3. **Update regularly**: React to new information
  4. **Avoid extremes**: Unless highly confident
  5. **Monitor performance**: Check calibration and scores
  6. **Document algorithm**: Help others understand your approach
  7. **Respect rate limits**: Don't overwhelm the API
</Tip>

<Warning>
  **Bot Etiquette:**

  * Don't create many redundant bots
  * Mark experimental bots as non-primary
  * Don't game the system with bot farms
  * Disclose bot methodology when asked
  * Don't use bots to manipulate community aggregates
  * Respect question author's bot inclusion preferences
</Warning>

## Bot Exclusion Filters

The codebase implements several bot filters:

### Query Filters

**Exclude Non-Primary Bots:**

```python theme={null}
forecasts.exclude_non_primary_bots()
```

Keeps:

* All human forecasters
* Primary bots (`is_primary_bot=True`)

Removes:

* Non-primary bots

**Exclude All Bots:**

```python theme={null}
forecasts.exclude(author__is_bot=True)
```

**Conditional Exclusion:**

```python theme={null}
base_forecasts = forecasts.exclude_non_primary_bots()

if not question.include_bots_in_aggregates:
    base_forecasts = base_forecasts.exclude(author__is_bot=True)
```

## Research Applications

Bots are valuable for research:

### Baseline Comparisons

* Simple statistical models as benchmarks
* Measure human value-add over algorithms
* Test "wisdom of crowds" threshold

### Hybrid Forecasting

* Combine human intuition + machine processing
* Test optimal human-bot mixing ratios
* Study when to trust humans vs algorithms

### Algorithm Development

* Rapidly test new forecasting methods
* Compare algorithm variants
* Build ensemble models

### Metaculus Research

* Study forecast updating patterns
* Analyze information integration
* Test aggregation methods
* Benchmark platform accuracy

## Technical Reference

**Key Files:**

* `users/services/bots_management.py` - Bot creation/management
* `users/models.py` - Bot user model fields
* `questions/models.py` - `include_bots_in_aggregates` field
* `scoring/score_math.py` - Bot scoring logic
* `utils/the_math/aggregations.py` - Bot inclusion in aggregates

**Database Fields:**

```python theme={null}
class User(AbstractUser):
    is_bot = models.BooleanField(default=False)
    is_primary_bot = models.BooleanField(default=False)
    bot_owner = models.ForeignKey('self', null=True, on_delete=models.CASCADE)

class Question(models.Model):
    include_bots_in_aggregates = models.BooleanField(default=False)
```

## Future Developments

Potential bot feature roadmap:

* Bot-specific scoring metrics
* Enhanced bot vs human leaderboards
* Bot methodology disclosure requirements
* Bot forecast explanations
* Collaborative human-bot forecasting tools
* Real-time bot performance dashboards