tiledb

Python interface to the TileDB storage engine

Showing:

64 Versions

0.11.3

TileDB-Py 0.11.3 Release Notes Impovements
  • Support for Python 3.10 #808
API Changes
  • Addition of tiledb.version() to return version as a tuple #801
  • Query.get_stats and Ctx.get_stats changed function signature; automatically print_out stats and add option to output as json #809
Bug fixes
  • tiledb.delete_fragments removes unused schemas #813

PyPI Wheels: https://pypi.org/project/tiledb/0.11.3/

0.11.2

TileDB-Py 0.11.2 Release Notes TileDB Embedded updates: Bug fixes
  • Support dict parameter for 'config' argument to VFS constructor #805

https://pypi.org/project/tiledb/0.11.2/

0.11.1

TileDB-Py 0.11.1 Release Notes TileDB Embedded updates: Bug Fixes
  • Correct libtiledb version checking for Fragment Info API getters' MBRs and array schema name #784

https://pypi.org/project/tiledb/0.11.1/

0.11.0

TileDB-Py 0.11.0 Release Notes TileDB Embedded updates: API Changes
  • Addition of MBRs to FragmentInfo API #760
  • Addition of array_schema_name to FragmentInfo API #777
  • Addition of tiledb.create_array_from_fragments to copy fragments within a given timestamp range to a new array #777

https://pypi.org/project/tiledb/0.11.0/

0.10.5

TileDB-Py 0.10.5 Release Notes API Changes
  • Addition of tiledb.delete_fragments to remove fragments within a given timestamp range #774

PyPI Wheels: https://pypi.org/project/tiledb/0.10.5/

0.10.4

TileDB-Py 0.10.4 Release Notes TileDB Embedded updates: Bug fixes
  • Error out when applying QueryCondition to dense arrays; this feature will be implemented in TileDB Embedded 2.5 #753
  • Ensure that indexer, multi-indexer, and .df return the same results when applying QueryCondition #753
  • Fix error when using .df with PyArrow 6 due to incorrect metadata field in exported schema #764
  • Fix #755: from_pandas to correctly round-trip unnamed Index #761
  • Fix .df indexer bug with empty result set #744
API Changes
  • Close the PyFragmentInfo object in the FragmentInfoList constructor to reflect changes in the FragmentInfo API in TileDB Embedded 2.5 #752
  • Make ctx argument optional for ArraySchemaEvolution #743
  • Remove coords_filters from ArraySchema for dense arrays #762

PyPI Wheels: https://pypi.org/project/tiledb/0.10.4/

0.10.3

TileDB-Py 0.10.3 Release Notes TileDB Embedded updates:
  • TileDB-Py 0.10.3 includes TileDB Embedded TileDB 2.4.2
    • Note that 2.4.1 was skipped due to accidental tagging of the 2.4.1 git tag during CI testing
API Changes
  • Addition of overwrite parameter to Array.create #713
  • Addition of "ascii" dtype for Dims #720
Bug fixes
  • Pass Ctx to ArraySchema.load in from_pandas #709
  • Give clear error message when attempting to apply QueryCondition on dimensions #722
  • Do not add string range when querying empty array #721
Improvements
  • String dimension default fix in core #2436 reverts a previous change in which the nonempty domain was passed the to multi-range indexer if unspecified #712

PyPI Wheels: https://pypi.org/project/tiledb/0.10.3/

0.10.2

TileDB-Py 0.10.2 Release Notes API Changes
  • Deprecate sparse writes to dense arrays #681
  • Addition of Attr.isascii #681
  • Addition of Ctx.get_stats and Query.get_stats #698
Improvements
  • Added support for timestamp argument in tiledb.from_numpy #699

PyPI Wheels: https://pypi.org/project/tiledb/0.10.2/

0.10.1

TileDB-Py 0.10.1 Release Notes API Changes
  • Do not require domain=(None, None) for string dimensions #662
Improvements Bug fixes
  • Fix concurrent use of Array.multi_index and .df by using new instance for each invocation #672
  • For attributes, if var=False but the bytestring is fixed-width or if var=True but the bytestring is variable length, error out #663

PyPI wheels: https://pypi.org/project/tiledb/0.10.1/

0.10.0

TileDB Embedded updates:
  • TileDB-Py 0.10.0 includes TileDB Embedded TileDB 2.4.0 with a number of new features and improvements, including:
    • new platform support: Apple M1
    • support for ArraySchema evolution (adding and removing attributes)
    • support for Azure SAS (shared access signature) tokens
API Changes
  • When using Array.multi_index, an empty result is returned if the nonempty domain is empty #656
  • Addition of Array.set_query to read array using a serialized query #651
Improvements
  • Support numeric column names in from_pandas by casting to str dtype #652
  • New tiledb.ArraySchemaEvolution API to add and drop attributes from an existing array #657
Bug Fixes
  • Correct listing of consolidated fragments to vacuum in the Fragment Info API by deprecating FragmentInfoList.to_vacuum_uri, FragmentInfoList.to_vacuum_num, FragmentInfo.to_vacuum_uri, and FragmentInfo.to_vacuum_num and replacing with FragmentInfoList.to_vacuum #650
  • Correct issue where appending None to FilterList causes segfault by checking the filter argument #653

PyPI wheels: https://pypi.org/project/tiledb/0.10.0/

0.9.5

TileDB-Py 0.9.5 Release Notes TileDB Embedded updates: Improvements
  • Consolidate _nonempty_domain_var into nonempty_domain #632
  • Support more valid Python syntax for QueryCondition statements #636
  • Addition of ascii dtype to Attr allows QueryCondition to support var-length strings #637

0.9.4

TileDB-Py 0.9.4 Release Notes Improvements
  • Support pickling for arrays in write-mode #626
Bug Fixes
  • Fixed multi-range indexer to default to explicitly pass in the non-empty domain if dimensions are unspecified #630

0.9.3

TileDB-Py 0.9.3 Release Notes

(intentionally left blank)

PyPI wheels: https://pypi.org/project/tiledb/0.9.3/

0.9.2

TileDB-Py 0.9.2 Release Notes Packaging Notes
  • Fixed release builder ordering issue which led to CRLF line endings in 0.9.1 source distribution.
API Changes
  • Deprecate Array.timestamp and replace with Array.timestamp_range #616
Improvements
  • Set ArraySchema.tile_order=None for Hilbert-ordered arrays #609
  • Use CIBW to build release wheels on Linux #613
  • Addition of Pickling functionality for SparseArray #618

PyPI wheels: https://pypi.org/project/tiledb/0.9.2

0.9.1

TileDB-Py 0.9.1 Release Notes Improvements
  • Support passing a timestamp range for consolidation and vacuuming #603
Bug Fixes
  • FragmentInfo API's to_vacuum_uri() function corrected to iterate through to_vacuum_num rather than fragment_num#603
  • Return "NA" For ArraySchema.tile_order if "Hilbert" #605

PyPI wheels: https://pypi.org/project/tiledb/0.9.1/

0.9.0

TileDB-Py 0.9.0 Release Notes TileDB Embedded updates:
  • TileDB-Py 0.9.0 includes TileDB Embedded TileDB 2.3 with a significant number of new features and improvements.
Packaging Notes
  • Windows wheels are now built with TileDB Cloud REST support enabled #541
Improvements
  • Addition of QueryCondition API to filter query on attributes #576
Bug Fixes
  • Fixed from_pandas append error for sparse arrayse: no need to specify 'row_start_idx' #593
  • Fixed 'index_dims' kwarg handling for from_pandas #590
API Changes
  • from_dataframe function has been removed; deprecated in TileDB-Py 0.6 and replaced by from_pandas.

PyPI wheels: https://pypi.org/project/tiledb/0.9.0/

0.8.11

TileDB-Py 0.8.11 Release Notes Bug fixes
  • Fixed incorrect NumPy ABI target in Linux wheels #590

(PyPI: https://pypi.org/project/tiledb/0.8.11/)

0.8.10

TileDB-Py 0.8.10 Release Notes Improvements
  • Disabled libtiledb Werror compilation argument for from-source builds via setup.py #574
  • Relaxed NumPy version requirements for from-source builds via setup.py #575
Bug fixes
  • Fix incorrect type for add_range with int8 dimension #573
    • Fixed FragmentInfoList where context was not being passed to ArraySchema #578
  • Fixed read bug due to large estimated result size #579
  • Fixed bug reading nullable attributes due to missing buffer resize #581
  • Fixed Python output for tiledb.stats_dump #586

0.8.9

TileDB-Py 0.8.9 Release Notes TileDB Embedded updates: Improvements
  • Support for iterating over incomplete query results #548
    • This feature provides the capability to consume partial query results with a fixed maximum buffer size rather than the the default behavior of resizing buffers and resubmitting to completion. Usage example: examples/incomplete_iteration.py (along with test in: test_libtiledb.py:test_incomplete_return)
  • Rename FragmentsInfo to FragmentInfoList #551
  • Dataframe creation uses Zstd default compression level (-1) #552
  • Rename Fragment Info API's non_empty_domain attribute to nonempty_domain #553
  • Added configuration option py.alloc_max_bytes to control maximum initial buffer allocation #557
Bug fixes
  • Fixed incorrected error raised in .df[] indexer when pyarrow not installed #554
  • Fixed from_pandas(attr_filters=None, dim_filters=None) (previously used internal defaults) #564
  • Fixed from_pandas write bug due to incorrect classification of str/bytes columns #562
  • Fix segfault due to mismatched validity num and data buffer sizes #567

0.8.8

TileDB-Py 0.8.8 Release Notes TileDB Embedded updates:

0.8.7

TileDB-Py 0.8.7 Release Notes Improvements
  • ArraySchema support for cell_order="hilbert" #535
Bug fixes
  • Fixed regression in from_pandas with string-valued index dimensions #526
  • Fixed GC lifetime bug in string buffer conversion #525
  • Fixed FilterList's repr() method #528

0.8.6

TileDB-Py 0.8.6 Release Notes TileDB Embedded updates: Improvements
  • Addition of VFS() functions copy_file() and copy_dir() #507
  • Add support in from_pandas for storing Pandas extension types as variable-length attributes #515
  • Add support for sparse writes to dense arrays #521
Bug fixes
  • Multi-length attributes, regardless of fixed or var-length, do not work query properly with PyArrow enabled due to lack of Arrow List support. When using .df[] with PyArrow enabled, we are returning a clear message to the user to use query(use_pyarrow=False) #513

0.8.5

TileDB-Py 0.8.5 Release Notes Documentation Updates
  • Added example reading/writing RGB (multi-component) array #487
Improvements
  • Restore tiledb.stats_dump default to verbose=True #491
  • Remove non_empty_domain_var() Fragment Info PyBind11 Function and only use get_non_empty_domain() for both fixed and var-length domains #505

0.8.4

TileDB-Py 0.8.4 Release Notes Improvements
  • Added support for from_pandas/df[] round-trip of Pandas nullable integer and bool types #480
  • Fragment info API example usage now provided at examples/fragment_info.py #479
  • fragment_info() is now exposed to the user and no longer has to be imported explicitly #479
  • fragment_info() parameters have been rearranged to match the rest of the TileDB Python API such that the uri is provided first and context, an optional parameter that defaults to tiledb.default_ctx(), is provided second #479
Bug fixes
  • Fix bug in Array.multi_index slicing bug for sparse array with dimension range including 0 #482

0.8.3

TileDB-Py 0.8.3 Release Notes TileDB Embedded updates: Improvements
  • Added nullable keyword argument to Attr constructor #474
Bug fixes
  • Fix bug in Array.multi_index with slice range including 0 (incorrectly used the nonempty domain as endpoint) #473

0.8.2

TileDB-Py 0.8.2 Release Notes Packaging Notes
  • This is a version bump to fix numpy compatibility pinning in the wheel build system.

0.8.1

TileDB-Py 0.8.1 Release Notes TileDB Embedded updates: Packaging Notes
  • TileDB-Py 0.8 does not support Python 2.

0.8.0

TileDB-Py 0.8.0 Release Notes TileDB Embedded updates:
  • TileDB-Py 0.8.0 includes TileDB Embedded 2.2.2 featuring a number of significant improvements in core storage engine functionality. See release notes for TileDB 2.2.1 and TileDB 2.2.2.
Packaging Notes
  • TileDB-Py 0.8 does not support Python 2.
Improvements
  • Add initial tiledb.from_parquet functionality (beta) [a90d5d9b1b]
  • Preload metadata in .df query path to reduce read latency for remote arrays [79ab12fcf0]
Bug fixes
  • Update py::dtype usage for compatibility with pybind11 2.6.2 [9d3d3d3c43]

0.7.7

TileDB-Py 0.7.7 Release Notes Bug fixes
  • Cherry-pick commit 9d3d3d3c43 to ix runtime bug in conda packages built against pybind11 2.6.2 9d3d3d3c430f

0.7.6

TileDB-Py 0.7.6 Release Notes Packaging Notes
  • TileDB-Py 0.7.x will be the last version of TileDB-Py supporting Python 2.
Bug fixes
  • Fix read compatibility for empty strings written with 2.1 or 2.2 #462
  • Fix #457: make sure to fit automatic tile extent to dim range for date type #464

0.7.5

TileDB-Py 0.7.5 Release Notes Packaging Notes
  • TileDB-Py 0.7.x will be the last version of TileDB-Py supporting Python 2.
TileDB Embedded updates: Improvements
  • FragmentInfo API by default returns information from all fragments and dimensions #444
  • Add integer multi-indexing for NumPy datetime64 dimensions #447
  • Add from_csv/pandas support for timestamp keyword argument to specify write timestamp #450
  • Add verbosity option to stats_dump() #452
  • Add unique_dim_values() to return unique dimension values for a given SparseArray #454
  • Add support to query() for returning subsets of specified dimensions #458
  • Optimize string array writes #459
Bug fixes
  • Fix Dim.shape for dense array with datetime dimension #448

0.7.4

TileDB-Py 0.7.4 Release Notes Improvements
  • Support selecting subset of dimensions in Array.query via new keyword argument dims: List[String]. The coords=True kwarg is still supported for compatibility, and continues to return all dimensions #433
  • Support Dim(filters=FilterList) keyword argument to set filters on a per-Dim basis #434
  • Support tiledb.from_csv setting attribute and dimension filters by dictionary of {name: filter} #434
  • Add ArraySchema.check wrapping tiledb_array_schema_check #435
  • Add support for attribute fill values tiledb.Attr(fill=...) and Attr.fill getter #437
API Changes
  • tiledb.from_csv keyword arg attrs_filters renamed to attr_filters #434
Bug fixes
  • Fix bug in multi_index slicing of dense arrays #438

0.7.3

TileDB-Py 0.7.3 Release Notes Improvements
  • The default result layout for indexing/querying sparse arrays is now TILEDB_UNORDERED #428, #431
  • Added documentation for all TileDB-Py configuration parameters #430
  • Fixed documentation rendering for Array.query #430
Bug fixes
  • Fix sparse dimension type selection when array type is not specified to from_pandas #429
  • Don't pass allows_duplicates=True to dense array constructor (tiledb.from_csv) #428

0.7.2

TileDB-Py 0.7.2 Release Notes TileDB Embedded updates: Changes
  • The default array type for from_pandas and from_csv is now dense, if unspecified, except when passing a dataframe with string indexes to from_pandas.
Improvements
  • Automatically determine column to dimension mapping for tiledb.from_csv append mode #408
  • Show var=True in Attr representation #423
Bug fixes
  • Fixed tiledb.from_csv/dataframe error when ingesting single-row/index datasets #422
  • Fixed intermittent csv_sparse_col_to_dims failure due to duplicate result ordering #423

0.7.1

TileDB-Py 0.7.1 Release Notes Improvements
  • Added support for df[] indexing via tiledb.Array.query #411
  • Modified stats_dump to return internal stats as string, allowing for output in Jupyter notebooks #403
  • Added __repr__ to Array and Ctx #413
  • tiledb.open now supports timestamp keyword argument #419
  • tiledb.Domain now supports passing a list of Dims without unpacking #419
Bug fixes
  • Fixed PyPI wheels load error on newer macOS due to overlinkage against system libraries in build process (curl -> libintl) #418
  • Fixed PyPI wheels load error on Windows due to building against TBB #419
  • Fixed indexing of attribute named 'coords' #414
  • open_dataframe now uses the underlying Array's nonempty_domain to avoid errors opening unlimited domain arrays #409

0.7.0

TileDB-Py 0.7.0 Release Notes

Wheels for this release are available on PyPI: https://pypi.org/project/tiledb/0.7.0/

TileDB Embedded updates:
  • TileDB-Py 0.7.0 includes TileDB Embedded 2.1.2 featuring a number of significant improvements, with major highlights including:
    • no longer uses Intel TBB for parallelization by default. Along with many benefits to TileDB Embedded, this significantly reduces complications and bugs with python multiprocessing fork mode.
    • Support coalescing subarray ranges to give major performance boosts.
Packaging Notes
  • TileDB-Py 0.7 packages on PyPI support macOS 10.13+ and manylinux10-compatible Linux distributions only. For now, wheels could be produced supporting older systems but without Google Cloud Support; if needed, please contact us to discuss.
Improvements
  • Added ".df[]" indexer tiledb.Array: directly returns a Pandas dataframe from a query (uses multi_index indexing behavior) #390
  • Added parallel CSV ingestion example using Python multiprocessing with tiledb.from_csv #397
  • Added wrapping and support for TileDB checksumming filters: ChecksumMD5Filter and ChecksumSHA256Filter #389
  • Removed TBB install from default setup.py, corresponding to TileDB Embedded changes #389
  • Add support for 'capacity' kwarg to from_csv/from_pandas #391
  • Add support for 'tile' kwarg to from_csv/from_pandas to customize Dim tile extent #391
  • Added '--release-symbols' option for building in release optimization with debug symbols #402
  • Changed allows_duplicates default to True for from_csv/from_pandas #394
Bug fixes
  • Fixed bug indexing anonymous attributes of sparse arrays using A[] (did not affect dense or multi_index) #404
  • Fixed rendering of column name in mixed dtype exception #382
  • Fixed forwarding of 'ctx' kwarg to from_csv/from_pandas #383
  • Fixed type of return values for empty results when indexing a sparse array #384
Misc Updates
  • Added round-trip tests for all filter repr objects #389

0.6.6

TileDB-Py 0.6.6 Release Notes

Note that we will be removing wheel support for macOS 10.9-10.12 in TileDB-Py 0.7 (planned for release in August 2020). This change is due to upstream (AWS SDK) minimum version requirements. The minimum supported version for macOS wheels on PyPI will be macOS 10.13.

Note that we will be removing support for manylinux1 wheels in TileDB-Py 0.7 (planned for release in August 2020). manylinux1 is based on CentOS5, which has been unsupported for several years. We now provide wheels built with manylinux2010, which is based on CentOS6 / glibc 2.12.

Improvements Bug fixes
  • Check for resizing offset buffers independently to fix query completion #377

0.6.5

TileDB-Py 0.6.5 Release Notes

We have added manylinux2010 wheels, corresponding to CentOS6 / glibc 2.12.

We are deprecating support for manylinux1 (CentOS5 / glibc 2.0.7), which is not supported by the Google Cloud Storage SDK. We are planning to remove manylinux1 wheel support in the TileDB-Py 0.7 release.

Improvements
  • Enabled Google Cloud Storage support in macOS and linux (manylinux2010) wheels on PyPI (#364)

0.6.4

TileDB-Py 0.6.4 Release Notes API notes
  • Deprecated initialize_ctx in favor of default_ctx(config: tiledb.Config) #351
Improvements
  • Bump release target to TileDB 2.0.6
  • Improved error reporting for input data type mismatches #359
  • Added tiledb.VFS.dir_size #343
  • Added read and buffer conversion statistics for python to tiledb.stats_dump #354
  • Implemented string deduplication to reduce conversion time for string arrays #357
Bug fixes
  • Fixed argument order for Array.consolidate with a Config override parameter #344

0.6.3

TileDB-Py 0.6.3 Release Notes Improvements Bug fixes
  • Fix unnecessary implicit ordering requirement for multi-attribute assignment. #328

0.6.2

TileDB-Py 0.6.2 Release Notes Bug fixes
  • Fix nonempty_domain with heterogeneous non-string dimensions (#320)
Improvements
  • Add doctest for tiledb.vacuum (#319)

0.6.1

TileDB-Py 0.6.1 Release Notes Improvements
  • Bump libtiledb version to 2.0.2.
Bug fixes
  • Fix assignment order for nonempty_domain with string dimensions (#308) (test in #311)
  • Fix bug in string attribute handling for var-length attributes (#307)
  • Fix regression reading anonymous attributes from TileDB 1.7 arrays (#311)
  • Fix incorrect multi_index error when string attribute results are empty (#311)

0.6.0

TileDB-Py 0.6.0 contains new functionality based on the significant new features in TileDB 2.0

  • Target TileDB version 2.0
    • Added support for heterogeneous and string-typed dimensions #304
      • The zipped coords return value for indexing results has been removed; coordinates for each dimension are now directly accessible by name.
    • Added support for tiledb_array_vacuum for cleaning up consolidated fragments
  • Added Windows wheels for Python 3.7 and 3.8 on PyPI

0.5.9

Improvements

0.5.8

  • Rebuild/release due to wheel build error on linux for 0.5.7.

0.5.7

TileDB-Py 0.5.7 Release Notes Improvements Bug fixes
  • Fix bug writing zero-length string (#291)

0.5.6

0.5.5

TileDB-Py 0.5.5 is a minor release featuring:

  • bump target libtiledb version to 1.7.4
  • return coordinates from Array.multi_index by default #259

0.5.4

TileDB-Py 0.5.4 is a minor release featuring:

  • bump target libtiledb version to 1.7.3
  • default Ctx initialization is now lazy #256

0.5.3

TileDB-Py 0.5.3 includes a number of improvements and bug fixes.

PyPI packages: https://pypi.org/project/tiledb/0.5.3/

Improvements
  • Reduce i/o overhead of tiledb.open and array constructors. #239, #240
  • Internal support for retrying incomplete queries in all array indexing modes. #238, #252
  • Eliminate reference cycles to improve Ctx cleanup. #249
  • Support for retrieving compressor level from filter. #234Bug fixes
  • multi_index result buffer calculation bug. #232
  • Variable-length indexing error. #236
  • Race condition initializing tiledb.cloud mixin from thread pool. #246

0.5.2

  • This release contains a bug-fix for multi_index queries #232

0.5.0

0.4.4

Bump release target to TileDB 1.6.3.

0.4.3

Bump TileDB core version to 1.6.

0.4.2

TileDB-Py 0.4.2 contains several improvements as well as bug-fixes associated with the TileDB 1.5.1 release.

New features
  • support for NumPy complex types (#142)
Bug fixes
  • fixed query buffer memory leak (#151)
  • fixed segfault during consolidation (TileDB #1213)
    • note: to receive this fix, conda and source builds should be updated to TileDB 1.5.1. TileDB-Py 0.4.2 binaries on PyPI bundle the updated TileDB 1.5.1 library.
  • fixed indexing with array dtype different from the platform default (#146)
  • fixed VFS.is_bucket when VFS is initialized with a Ctx object (#148)
  • fixed schema_like to correctly forward a Ctx keyword arg (#148)

0.4.1

TileDB-Py 0.4.1 features:

0.4.0

This release builds TileDB-Py against TileDB 1.5

Breaking changes:

Major features of TileDB-Py 0.4 include:

0.3.1

0.3.0

0.2.1

0.2.0

0.1.1

0.1.1.dev2