inferactively-pymdp
pypi i inferactively-pymdp

inferactively-pymdp

A Python implementation of active inference for Markov Decision Processes

by infer-actively

0.0.5 (see all)License:MIT
pypi i inferactively-pymdp
Readme

A Python package for simulating Active Inference agents in Markov Decision Process environments. Please see our companion preprint on arxiv for an overview of the package, its structure, and a mathematical overview of active inference in partially-observable Markov Decision Processes: "pymdp: A Python library for active inference in discrete state spaces".

This package is hosted on the infer-actively GitHub organization, which was built with the intention of hosting open-source active inference and free-energy-principle related software.

Most of the low-level mathematical operations are NumPy ports of their equivalent functions from the SPM implementation in MATLAB. We have benchmarked and validated most of these functions against their SPM counterparts.

Status

status PyPI version Documentation Status DOI

pymdp in action

Here's a visualization of pymdp agents in action. One of the defining features of active inference agents is the drive to maximize "epistemic value" (i.e. curiosity). Equipped with such a drive in environments with uncertain yet disclosable hidden structure, active inference can ultimately allow agents to simultaneously learn about the environment as well as maximize reward.

The simulation below (see associated notebook here) demonstrates what might be called "epistemic chaining," where an agent (here, analogized to a mouse seeking food) forages for a chain of cues, each of which discloses the location of the subsequent cue in the chain. The final cue (here, "Cue 2") reveals the location a hidden reward. This is similar in spirit to "behavior chaining" used in operant conditioning, except that here, each successive action in the behavioral sequence doesn't need to be learned through instrumental conditioning. Rather, active inference agents will naturally forage the sequence of cues based on an intrinsic desire to disclose information. This ultimately leads the agent to the hidden reward source in the fewest number of moves as possible.

You can run the code behind simulating tasks like this one and others in the Examples section of the official documentation.


Cue 2 in Location 1, Reward on Top


Cue 2 in Location 3, Reward on Bottom

Quick-start: Installation and Usage

In order to use pymdp to build and develop active inference agents, we recommend installing it with the the package installer pip, which will install pymdp locally as well as its dependencies. This can also be done in a virtual environment (e.g. with venv).

When pip installing pymdp, use the package name inferactively-pymdp:

pip install inferactively-pymdp

Once in Python, you can then directly import pymdp, its sub-packages, and functions.


import pymdp
from pymdp import utils
from pymdp.agent import Agent

num_obs = [3, 5] # observation modality dimensions
num_states = [3, 2, 2] # hidden state factor dimensions
num_controls = [3, 1, 1] # control state factor dimensions
A_matrix = utils.random_A_matrix(num_obs, num_states) # create sensory likelihood (A matrix)
B_matrix = utils.random_B_matrix(num_states, num_controls) # create transition likelihood (B matrix)

C_vector = utils.obj_array_uniform(num_obs) # uniform preferences

# instantiate a quick agent using your A, B and C arrays
my_agent = Agent( A = A_matrix, B = B_matrix, C = C_vector)

# give the agent a random observation and get the optimized posterior beliefs

observation = [1, 4] # a list specifying the indices of the observation, for each observation modality

qs = my_agent.infer_states(observation) # get posterior over hidden states (a multi-factor belief)

# Do active inference

q_pi, neg_efe = my_agent.infer_policies() # return the policy posterior and return (negative) expected free energies of each policy as well

action = my_agent.sample_action() # sample an action

# ... and so on ...

Getting started / introductory material

We recommend starting with the Installation/Usage section of the official documentation for the repository, which provides a series of useful pedagogical notebooks for introducing you to active inference and how to build agents in pymdp.

For new users to pymdp, we specifically recommend stepping through following three Jupyter notebooks (can also be used on Google Colab):

Special thanks to Beren Millidge and Daphne Demekas for their help in prototyping earlier versions of the Active Inference from Scratch tutorial, which were originally based on a grid world POMDP environment create by Alec Tschantz.

We also have (and are continuing to build) a series of notebooks that walk through active inference agents performing different types of tasks, such as the classic T-Maze environment and the newer Epistemic Chaining demo.

Contributing

This package is under active development. If you would like to contribute, please refer to this file

If you would like to contribute to this repo, we recommend using venv and pip

cd <path_to_repo_fork>
python3 -m venv env
source env/bin/activate
pip install -r requirements.txt
pip install -e ./ # This will install pymdp as a local dev package

You should then be able to run tests locally with pytest

pytest test

Authors

GitHub Stars

152

LAST COMMIT

7mos ago

MAINTAINERS

1

CONTRIBUTORS

13

OPEN ISSUES

4

OPEN PRs

3
VersionTagPublished
0.0.5
4mos ago
0.0.4
7mos ago
0.0.3
10mos ago
0.0.2
1yr ago
No alternatives found
No tutorials found
Add a tutorial