Commit Graph

12 Commits

Author SHA1 Message Date
Dan Blanchard 324e3c9419 ENH sort option for memory-efficient DictVectorizer
Added memory efficient UnsortedDictVectorizer

When loading really large files for SKLL, I found that temporarily
storing a list of dictionaries to pass to DictVectorizer was frequently
using up huge amounts of memory. You can now call `fit_transform` on an
iterable, and not have to waste the temporary space.

Sorting is done in-place, so it doesn't waste memory.
2014-09-22 16:44:31 +02:00
Lars Buitinck 89d94ca987 TST older nosetests compat in DictVectorizer test 2014-02-23 18:38:10 +01:00
Lars Buitinck bd0cf75afa FIX DictVectorizer handling of empty inputs
Fixes #2883.
2014-02-23 18:03:50 +01:00
Lars Buitinck e163f8bfef FIX DictVectorizer behavior on empty X and empty samples
More useful error message for former; return all zeros for latter.

Also changed integer type to np.intc, which matches scipy.sparse
index types more closely.

Fixes #1903.
2013-05-04 17:09:07 +02:00
Robert Layton dacfd8bd5d DOC: Replaced all BSD style licenses with "BSD 3 clause"
Replaced all BSD style licenses with "BSD 3 clause"
Not checked yet!

Removed duplicate "3 clause, 3 clause"

Removed trailing period if exists

Fixed some missed licences, still about 50 to do, but those can be automated

Think I got the last of them.

Apparently me and sed have different ideas of regex.

Found a few more
2013-04-30 08:34:46 +02:00
Olivier Grisel cde9fbf9c4 P3K: support for py3k in dict_vectorizer module 2013-02-03 20:04:19 +01:00
Lars Buitinck 749ea70370 ENH let DictVectorizer build a CSR matrix directly and use array.array
Prevents copying.
2012-09-29 23:01:27 +02:00
Olivier Grisel e4ce5cf20d ENH: sort features in dict vectorizer + new doc 2012-04-01 03:13:23 +02:00
Lars Buitinck 7af5f6a549 BUG fix doctests for DictVectorizer (nose 0.X compat) 2012-03-17 14:52:24 +01:00
Lars Buitinck e05e1a42b9 ENH reduce memory usage of DictVectorizer.transform in sparse case
Don't materialize iterable's entire contents.
2012-03-17 12:18:22 +01:00
Lars Buitinck f3263661f5 TEST more strict test for one-of-K coding in DictVectorizer 2012-03-17 01:19:48 +01:00
Lars Buitinck c128b147b4 COSMIT rename DictVectorizer source files
At the request of @mblondel.
2012-03-16 23:12:41 +01:00