Python package for scraping personal financial data from financial institutions.
This package may be useful on its own, but is specifically designed to be used with beancount-import.
To install the most recent published package from PyPi, simply type:
pip install finance-dl
To install from a clone of the repository, type:
pip install .
or for development:
pip install -e .
Create a Python file like
Refer to the documentation of the individual scraper modules for details.
You can run a scraping configuration named
myconfig as follows:
python -m finance_dl.cli --config-module example_finance_dl_config --config myconfig
myconfig refers to a function named
CONFIG_myconfig in the configuration module.
Make sure that your configuration module is accessible in your Python
sys.path includes the current directory by
default, you can simply run this command from the directory that
contains your configuration module.
By default, the scrapers run fully automatically, and the ones based
chromedriver run in headless mode. If the initial
attempt for a
selenium-based scraper fails, it is automatically
retried again with the browser window visible. This allows you to
manually complete the login process and enter any multi-factor
authentication code that is required.
To debug a scraper, you can run it in interactive mode by specifying
-i command-line argument. This runs an interactive IPython
shell that lets you manually invoke parts of the scraping process.
To run multiple configurations at once, and keep track of when each
configuration was last updated, you can use the
To display the update status, first create a
logs directory and run:
python -m finance_dl.update --config-module example_finance_dl_config --log-dir logs status
Initially, this will indicate that none of the configurations have
been updated. To update a single configuration
python -m finance_dl.update --config-module example_finance_dl_config --log-dir logs update myconfig
With a single configuration specified, this does the same thing as the
finance_dl.cli tool, except that the log messages are written to
logs/myconfig.txt and a
logs/myconfig.lastupdate file is created
if it is successful.
If multiple configurations are specified, as in:
python -m finance_dl.update --config-module example_finance_dl_config --log-dir logs update myconfig1 myconfig2
then all specified configurations are run in parallel.
To update all configurations, run:
python -m finance_dl.update --config-module example_finance_dl_config --log-dir logs update --all
Copyright (C) 2014-2018 Jeremy Maitin-Shepard.
Distributed under the GNU General Public License, Version 2.0 only. See LICENSE file for details.