Skip to content

IndexDownloadHelper

IndexDownloadHelper is a Python package that simplifies downloading financial index and constituent data from the LSEG Data API.

Disclaimer

This project is not affiliated with, endorsed by, or sponsored by the London Stock Exchange Group (LSEG).

It is an independent wrapper built on top of the official lseg-data Python package. Users must have valid credentials and licenses to use LSEG data services.

Please note: the source code cannot be distributed at this time.

Features

The package simplifies the process of retrieving historical financial data by automatically managing:

  • Rate limits, dynamic retries, and API delays
  • Data batching and chunking to stay within per-request data point limits
  • Storing extracted data systematically into dynamically created local CSV structures

Documentation

Read the documentation on GitLab Pages.

Installation

Prerequisites

  • Python 3.10+
  • Active access to the LSEG Workspace (formerly Refinitiv Workspace) API with properly configured application credentials.

Using setup.py

Note: the source code is not publicly distributed at present. The steps below apply to authorized users only.

Clone the repository and install the package locally:

git clone https://github.com/trholy/indexdownloadhelper
cd indexdownloadhelper
pip install -e .

Example usage

The example_usage directory contains scripts that walk through a full end-to-end workflow: downloading an index chain, its fundamentals, fundamental data for the constituents, and historical time series.

The entire process is orchestrated by run_manager.py, which loads configurations from a local .env file and runs the following scripts sequentially:

1. Download the index chain (download_index_chain.py)

Fetches the constituent mapping of an index for a set of target dates.

  • Inputs: the list of indices, the LSEG API fields to fetch, and the generated target dates.

  • Output: Saves the index chain structure to download/<index_name>/index_chain/.

2. Download time series (download_time_series.py)

Downloads historical time series for the index itself and for every constituent identified in the index chains collected in step 1.

  • Inputs: the index list, start and end dates, fields (daily close price, for example), the interval, and predefined chunk sizes.

  • Output: Stitches and saves continuous time series data to download/<index_name>/time_series_data/.

3. Filter valid dates (FileHandler.filter_valid_dates)

Before running the remaining steps for each index, run_manager.py validates which target dates actually have non-empty chain files with constituent RICs.

  • Inputs: index_name, the generated target dates, and the expected chain column (Constituent RIC).

  • Behavior: if no valid dates are found for an index, steps 4 to 6 are skipped for it.

4. Download index fundamentals (download_index_fundamentals.py)

Downloads index- or ETP-level constituent-weight fundamentals for each valid date.

  • Inputs: the per-index list of valid dates, the NaN threshold, and fields selected automatically according to the target type.

  • Field selection logic:

  • ETP → TR.ETPConstituentRIC, TR.ETPConstituentWeightPercent

  • INDEX (or fallback) → TR.IndexConstituentRIC, TR.IndexConstituentWeightPercent

  • Details: data quality is checked against a configurable threshold for the share of NaN values.

  • Output: Saves results to download/<index_name>/additional_index_data/.

5. Download constituent fundamentals (download_constituents_fundamentals.py)

Downloads further fundamental data (ESG scores, market capitalization, and so on) for each unique constituent on the target dates.

  • Inputs: the index list, the unique RICs (parsed locally from the index chain files), the data fields, and the target dates.

  • Output: each field is downloaded separately in batches and the results are saved as a single combined DataFrame under download/<index_name>/additional_stock_data/.

6. Download the index info snapshot (download_index_info.py)

Downloads additional point-in-time metrics at index level, but only for the most recent valid date (sorted(valid_dates)[-1]).

  • Inputs: the index list, the name mapping, a single target date, and the index info fields.

  • Output: Saves results under download/<index_name>/additional_index_info/.

Configuration via .env

run_manager.py reads its parameters from a .env file. The main variables are:

  • INDICES_LIST: JSON array of indices/ETPs.
  • NAME_MAPPING_DICT: JSON object mapping human-readable names to specific RICs.
  • TARGET_DATES_PARAMS: JSON object with start_date, end_date, and frequency, from which the chronological array of target dates is generated.
  • DATA_FIELDS_CHAIN: the fields for the index constituent mapping.
  • DATA_FIELDS_INFO: the fields for index info snapshot downloads.
  • FIELDS_TIME_SERIES: the target fields for time series history.
  • TIMESERIES_INTERVAL: the granularity of the history (for example 'daily').
  • DATA_FIELDS_CONST_FUNDAMENTAL: the fields for cross-sectional constituent fundamentals.
  • NAN_THRESHOLD: the maximum tolerated share of NaN values (0–1) before a warning is raised for incomplete asset data.

Execution

Make sure your Python environment is authenticated against the LSEG workspace, fill in the .env file with the variables above, and run the orchestrator:

python example_usage/run_manager.py

This single command works through the configurations in order, queries the LSEG Data API, respects its rate limits, and writes the CSV files into the download/ working directory.