> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/Metaculus/metaculus/llms.txt
> Use this file to discover all available pages before exploring further.

# Scores

> Access detailed scoring and performance data

## Overview

The Scores endpoints provide detailed information about how forecasts are scored on Metaculus. Scores measure forecast accuracy using proper scoring rules and are the basis for leaderboards and performance tracking.

<Note>
  Scores are typically accessed through leaderboard endpoints and post data downloads. These endpoints provide raw scoring data for advanced analysis.
</Note>

***

## Understanding Metaculus Scoring

### Score Types

Metaculus uses several scoring methods to evaluate forecasts:

<ResponseField name="peer" type="Score Type">
  **Peer Score**: Measures performance relative to the community aggregate prediction. Rewards forecasters who beat the crowd.

  Calculated as: Your log score - Community log score
</ResponseField>

<ResponseField name="baseline" type="Score Type">
  **Baseline Score**: Measures performance relative to a simple baseline prior (e.g., uniform distribution for continuous questions, 50% for binary).

  More generous than peer score, useful for beginners.
</ResponseField>

<ResponseField name="spot_peer" type="Score Type">
  **Spot Peer Score**: Peer score evaluated at a specific time (CP reveal time) rather than continuously weighted.

  Used for tournament scoring to prevent gaming through frequent updates.
</ResponseField>

<ResponseField name="spot_baseline" type="Score Type">
  **Spot Baseline Score**: Baseline score evaluated at CP reveal time.
</ResponseField>

<ResponseField name="relative_legacy" type="Score Type">
  **Legacy Relative Score**: Historical scoring method from old Metaculus. Deprecated.
</ResponseField>

### Scoring Mechanics

<Note>
  **How Scores Are Calculated**

  1. **Log Score**: Your forecast is scored using logarithmic scoring rules. For binary questions with outcome O and prediction p:
     * Score = log₂(p) if O = Yes
     * Score = log₂(1-p) if O = No

  2. **Continuous Questions**: CDF is converted to PMF, then log score is calculated based on the probability mass assigned to the actual outcome.

  3. **Coverage**: Your score is weighted by what fraction of scored questions you forecasted. Higher coverage = more reliable score.

  4. **Aggregation**: Scores across questions are averaged with question weights applied.
</Note>

***

## Download Score Data

Score data is primarily accessed through the data download endpoints:

### Post-Level Scores

<CodeGroup>
  ```bash cURL theme={null}
  curl -X GET "https://www.metaculus.com/api/posts/3530/download-data/?include_scores=true" \
    -H "Authorization: Token YOUR_TOKEN" \
    --output question_data.zip
  ```

  ```python Python theme={null}
  import requests
  import pandas as pd
  import zipfile
  import io

  response = requests.get(
      "https://www.metaculus.com/api/posts/3530/download-data/",
      headers={"Authorization": "Token YOUR_TOKEN"},
      params={"include_scores": True}
  )

  # Extract and read score data
  with zipfile.ZipFile(io.BytesIO(response.content)) as zf:
      with zf.open('score_data.csv') as f:
          scores_df = pd.read_csv(f)

  print(scores_df.head())
  print(f"\nTotal scores: {len(scores_df)}")
  ```
</CodeGroup>

See the [Posts](/api/posts) endpoint documentation for full details on the download-data endpoint.

### Project-Level Scores

<CodeGroup>
  ```bash cURL theme={null}
  curl -X GET "https://www.metaculus.com/api/projects/144/download-data/?include_scores=true" \
    -H "Authorization: Token YOUR_TOKEN" \
    --output project_data.zip
  ```

  ```python Python theme={null}
  import requests
  import pandas as pd
  import zipfile
  import io

  response = requests.get(
      "https://www.metaculus.com/api/projects/144/download-data/",
      headers={"Authorization": "Token YOUR_TOKEN"},
      params={"include_scores": True}
  )

  with zipfile.ZipFile(io.BytesIO(response.content)) as zf:
      with zf.open('score_data.csv') as f:
          scores_df = pd.read_csv(f)

  print(f"Questions with scores: {scores_df['Question ID'].nunique()}")
  print(f"Users with scores: {scores_df['User ID'].nunique()}")
  ```
</CodeGroup>

See the [Projects](/api/projects) endpoint documentation for details.

***

## Score Data Format

When you download score data, you receive a CSV with the following schema:

### Score Data CSV Schema

<ResponseField name="Question ID" type="integer">
  The question ID this score is for
</ResponseField>

<ResponseField name="User ID" type="integer">
  The user ID who earned this score
</ResponseField>

<ResponseField name="User Username" type="string">
  The username of the scorer
</ResponseField>

<ResponseField name="Score Type" type="string">
  Type of score: `peer`, `baseline`, `spot_peer`, `spot_baseline`, `relative_legacy`, or `manual`
</ResponseField>

<ResponseField name="Score" type="number">
  The score value. Higher is better. Can be negative.
</ResponseField>

<ResponseField name="Coverage" type="number">
  The coverage value (0-1) representing what fraction of time the user had an active forecast
</ResponseField>

***

## Accessing Scores in Aggregations

Scores for community aggregations are included in question data when using `with_cp=true`:

<CodeGroup>
  ```python Get Aggregation Scores theme={null}
  import requests

  response = requests.get(
      "https://www.metaculus.com/api/posts/3530/",
      headers={"Authorization": "Token YOUR_TOKEN"},
      params={"with_cp": True}
  )

  post = response.json()
  question = post["question"]
  aggregations = question["aggregations"]

  # Get recency-weighted aggregation scores
  rw_scores = aggregations["recency_weighted"]["score_data"]

  print("Recency Weighted Aggregation Scores:")
  print(f"  Peer Score: {rw_scores.get('peer_score', 'N/A')}")
  print(f"  Baseline Score: {rw_scores.get('baseline_score', 'N/A')}")
  print(f"  Coverage: {rw_scores.get('coverage', 'N/A')}")
  ```
</CodeGroup>

***

## Example: Analyze User Performance

```python theme={null}
import requests
import pandas as pd
import zipfile
import io

# Download scores for a project
response = requests.get(
    "https://www.metaculus.com/api/projects/3876/download-data/",
    headers={"Authorization": "Token YOUR_TOKEN"},
    params={"include_scores": True}
)

with zipfile.ZipFile(io.BytesIO(response.content)) as zf:
    with zf.open('score_data.csv') as f:
        scores_df = pd.read_csv(f)
    with zf.open('question_data.csv') as f:
        questions_df = pd.read_csv(f)

# Filter to peer scores only
peer_scores = scores_df[scores_df['Score Type'] == 'peer']

# Calculate statistics per user
user_stats = peer_scores.groupby('User Username').agg({
    'Score': ['mean', 'sum', 'count'],
    'Coverage': 'mean'
}).round(2)

user_stats.columns = ['Avg Score', 'Total Score', 'Questions', 'Avg Coverage']
user_stats = user_stats.sort_values('Total Score', ascending=False)

print("Top 10 Forecasters by Total Peer Score:")
print(user_stats.head(10))

# Analyze by question type
question_types = questions_df.set_index('Question ID')['Question Type']
scores_with_type = peer_scores.copy()
scores_with_type['Question Type'] = scores_with_type['Question ID'].map(question_types)

type_stats = scores_with_type.groupby('Question Type')['Score'].agg(['mean', 'count'])
print("\nAverage Score by Question Type:")
print(type_stats)
```

***

## Example: Coverage Analysis

```python theme={null}
import requests
import pandas as pd
import matplotlib.pyplot as plt
import zipfile
import io

response = requests.get(
    "https://www.metaculus.com/api/posts/3530/download-data/",
    headers={"Authorization": "Token YOUR_TOKEN"},
    params={"include_scores": True}
)

with zipfile.ZipFile(io.BytesIO(response.content)) as zf:
    with zf.open('score_data.csv') as f:
        scores_df = pd.read_csv(f)

# Filter to peer scores
peer_scores = scores_df[scores_df['Score Type'] == 'peer'].copy()

# Create coverage bins
peer_scores['Coverage Bin'] = pd.cut(
    peer_scores['Coverage'],
    bins=[0, 0.25, 0.5, 0.75, 1.0],
    labels=['0-25%', '25-50%', '50-75%', '75-100%']
)

# Analyze score by coverage
coverage_stats = peer_scores.groupby('Coverage Bin')['Score'].agg(['mean', 'std', 'count'])

print("Score Statistics by Coverage:")
print(coverage_stats)

# Plot
plt.figure(figsize=(10, 6))
plt.bar(range(len(coverage_stats)), coverage_stats['mean'], 
        yerr=coverage_stats['std'], capsize=5)
plt.xlabel('Coverage Bin')
plt.ylabel('Average Peer Score')
plt.title('Forecast Accuracy vs Coverage')
plt.xticks(range(len(coverage_stats)), coverage_stats.index)
plt.grid(axis='y', alpha=0.3)
plt.tight_layout()
plt.savefig('coverage_analysis.png')
```

***

## Example: Historical Score Tracking

```python theme={null}
import requests
import pandas as pd
import zipfile
import io
from datetime import datetime

# Download question and forecast data
response = requests.get(
    "https://www.metaculus.com/api/posts/3530/download-data/",
    headers={"Authorization": "Token YOUR_TOKEN"},
    params={"include_scores": True}
)

with zipfile.ZipFile(io.BytesIO(response.content)) as zf:
    with zf.open('forecast_data.csv') as f:
        forecasts_df = pd.read_csv(f)
    with zf.open('score_data.csv') as f:
        scores_df = pd.read_csv(f)
    with zf.open('question_data.csv') as f:
        questions_df = pd.read_csv(f)

# Focus on a specific user
my_user_id = 12345

my_forecasts = forecasts_df[
    (forecasts_df['Forecaster ID'] == my_user_id) &
    (forecasts_df['Forecaster ID'].notna())  # Exclude aggregations
].copy()

my_scores = scores_df[
    (scores_df['User ID'] == my_user_id) &
    (scores_df['Score Type'] == 'peer')
].copy()

# Merge to get forecast timestamps
my_forecasts['Start Time'] = pd.to_datetime(my_forecasts['Start Time'])
my_forecasts['Month'] = my_forecasts['Start Time'].dt.to_period('M')

# Calculate rolling performance
monthly_questions = my_forecasts.groupby('Month')['Question ID'].nunique()
monthly_forecasts = my_forecasts.groupby('Month').size()

print(f"Your Forecasting Activity:")
print(f"  Questions forecasted: {my_forecasts['Question ID'].nunique()}")
print(f"  Total forecasts: {len(my_forecasts)}")
print(f"  Average peer score: {my_scores['Score'].mean():.2f}")
print(f"  Score std dev: {my_scores['Score'].std():.2f}")
print(f"\nMonthly Activity:")
for month, count in monthly_questions.tail(6).items():
    forecasts = monthly_forecasts[month]
    print(f"  {month}: {count} questions, {forecasts} forecasts")
```

***

## Important Notes

<Warning>
  **Score Data Access**

  Individual user scores are only available:

  * To the user themselves
  * To site administrators
  * In aggregate form (leaderboards)

  You cannot download other users' detailed score data for privacy reasons.
</Warning>

<Note>
  **Score Timing**

  Scores are calculated:

  * When questions resolve
  * When leaderboards are updated (typically daily)
  * When explicitly recalculated by admins

  There may be a delay between question resolution and score appearance.
</Note>

<Note>
  **Coverage Matters**

  High coverage (forecasting many questions) makes scores more reliable and statistically meaningful. Users with low coverage may have high variance in their scores.
</Note>