87a6a337aa
write a dump exporter
2022-10-27 11:33:51 +02:00
2c8f1a43e9
get rids of meilisearch-lib
2022-10-27 11:33:44 +02:00
8770e07397
I can index documents without meilisearch
2022-10-27 11:33:42 +02:00
94e29a9f5f
extract the index abstraction out of the index-scheduler in its own module
2022-10-27 11:33:38 +02:00
a9844bd4f6
move the update file store to another crate with as little dependencies as possible
2022-10-27 11:33:37 +02:00
1a47949063
START THE REWRITE OF THE INDEX SCHEDULER: index & register has been implemented
2022-10-27 11:33:33 +02:00
98fc093823
Optimize a few performance sensitive dependencies on debug builds
2022-10-12 09:22:05 +02:00
5cfb5df31e
Set opt-level to 0 for debug builds
...
But speed up compile times by optimising build dependencies of lindera
2022-10-12 09:22:05 +02:00
a3622eda46
Merge #642
...
642: Remove LTO in release profile r=Kerollmops a=loiclec
Since we can't enable it in Meilisearch (see https://github.com/meilisearch/meilisearch/pull/2717 ), we should not enable it in milli either. The goal is for milli's benchmarks to accurately represent its performance within meilisearch.
Co-authored-by: Loïc Lecrenier <loic@meilisearch.com>
2022-09-21 09:14:46 +00:00
513a38f07b
Remove LTO in release profile
...
Since we can't enable it in Meilisearch, there is no point in having it
enabled in milli
2022-09-21 10:44:33 +02:00
b6fe6838d3
Remove `helpers` crate
2022-09-21 10:25:36 +02:00
add96f921b
Remove unused infos/ http-ui/ and fuzz/ crates
2022-09-14 06:55:01 +02:00
9a789daa58
Disable LTO due to compilation failures on some platforms
...
(aarch64 linux)
2022-08-29 17:21:08 +02:00
72f8adaa70
Add LTO and codegen-units=1 to release compile options
2022-08-23 13:03:57 +02:00
03e679b634
Make binaries faster on release profile through better compile options
2022-08-17 10:29:33 +02:00
36cb09eb25
Add a new `meilisearch_types` crate
...
Move `meilisearch_error` to `meilisearch_types::error`
Move `meilisearch_lib::index_resolver::IndexUid` to `meilisearch_types::index_uid`
Add a new `InvalidIndexUid` error in `meilisearch_types::index_uid`
2022-06-09 16:14:13 +02:00
5f0e9b63d2
chore(dump): add tests
2022-05-24 14:21:56 +02:00
1ef87cc6d0
chore: move permissive-json-pointer in the meilisearch repository
...
Update permissive-json-pointer/src/lib.rs
Co-authored-by: Clément Renault <clement@meilisearch.com>
2022-04-20 19:24:41 +02:00
c2469b6765
create the json-depth-checker crate
2022-04-14 11:14:08 +02:00
bab898ce86
move the flatten-serde-json crate inside of milli
2022-04-07 18:20:44 +02:00
2a31cd13c9
set resolver to v2
2022-03-16 11:47:27 +01:00
748d5b69a5
cargo: use resolver 2
2022-03-14 23:18:59 -04:00
3ecebd15ee
chore(all): fix rust edition
2022-01-12 11:14:50 +01:00
5fb4ed60e7
chore(all) set rust edition to 2021
2022-01-06 13:30:45 +01:00
ffefd0caf2
feat(auth): API keys
...
implements:
https://github.com/meilisearch/specifications/blob/develop/text/0085-api-keys.md
- Add tests on API keys management route (meilisearch-http/tests/auth/api_keys.rs)
- Add tests checking authorizations on each meilisearch routes (meilisearch-http/tests/auth/authorization.rs)
- Implement API keys management routes (meilisearch-http/src/routes/api_key.rs)
- Create module to manage API keys and authorizations (meilisearch-auth)
- Reimplement GuardedData to extend authorizations (meilisearch-http/src/extractors/authentication/mod.rs)
- Change X-MEILI-API-KEY by Authorization Bearer (meilisearch-http/src/extractors/authentication/mod.rs)
- Change meilisearch routes to fit to the new authorization feature (meilisearch-http/src/routes/)
- close #1867
2021-12-06 09:52:41 +01:00
a30e02c18c
feat(all): Task store
...
implements:
https://github.com/meilisearch/specifications/blob/develop/text/0060-refashion-updates-apis.md
linked PR:
- #1889
- #1891
- #1892
- #1902
- #1906
- #1911
- #1914
- #1915
- #1916
- #1918
- #1924
- #1925
- #1926
- #1930
- #1936
- #1937
- #1942
- #1944
- #1945
- #1946
- #1947
- #1950
- #1951
- #1957
- #1959
- #1960
- #1961
- #1962
- #1964
- https://github.com/meilisearch/milli/pull/414
- https://github.com/meilisearch/milli/pull/409
- https://github.com/meilisearch/milli/pull/406
- https://github.com/meilisearch/milli/pull/418
- close #1687
- close #1786
- close #1940
- close #1948
- close #1949
- close #1932
- close #1956
2021-12-02 20:14:42 +01:00
f28600031d
Rename the filter_parser crate into filter-parser
...
Co-authored-by: Clément Renault <clement@meilisearch.com>
2021-11-09 16:41:10 +01:00
7e5c5c4d27
start a new rewrite of the filter parser
2021-10-22 01:15:42 +02:00
14c6ae4735
disable stripping
2021-10-07 12:10:36 +02:00
493a0e377d
optimize build size
2021-10-07 11:49:52 +02:00
5fad37aebd
Merge #1711
...
1711: MeiliSearch refactor introducing OBKV format r=MarinPostma a=MarinPostma
This PR refactor some multiple components of meilisearch, and introduce the obkv document format to meilisearch
- [x] Split meilisearch-http and meilisearch-lib
- [x] Replace `IndexActor` and `UuidResolver` with `IndexResolver`
- [x] Remove mentions to Actor
- [x] Remove Actor traits to simplify code
- [x] Integrate obkv document format
- [x] Remove `Data`
- [x] Restore all route
- [x] Replace `Box<dyn error>` with `anyhow::Error`
- [x] Introduce update file store
- [x] Update file store error handling
- [x] Fix dumps
- [x] Fix snapshots
- [x] Fix tests
- [x] Update module documentation
- [x] add csv suppport (feat `@ManyTheFish` #1729 )
- [x] add jsonl support
- [x] integrate geosearch (feat `@irevoire` #1725 )
partially implements #1691 and #1690 . The error handling is very basic now, I will finish it in the next pr.
Some unit tests have been disabled, I will re-enable them ASAP, but they need a bit more work.
close #1531
P.S: sorry for this monstrous PR :'(
Co-authored-by: mpostma <postma.marin@protonmail.com>
Co-authored-by: Tamo <tamo@meilisearch.com>
Co-authored-by: many <maxime@meilisearch.com>
2021-09-29 14:38:55 +00:00
aa6c5df0bc
Implement documents format
...
document reader transform
remove update format
support document sequences
fix document transform
clean transform
improve error handling
add documents! macro
fix transform bug
fix tests
remove csv dependency
Add comments on the transform process
replace search cli
fmt
review edits
fix http ui
fix clippy warnings
Revert "fix clippy warnings"
This reverts commit a1ce3cd96e603633dbf43e9e0b12b2453c9c5620.
fix review comments
remove smallvec in transform loop
review edits
2021-09-21 16:58:33 +02:00
60518449fc
split meilisearch-http and meilisearch-lib
2021-09-21 13:23:22 +02:00
770b6d25ae
deps: unify pest dependency
2021-09-15 12:15:44 +09:00
41bdc90f46
Revert "Enable optimization in every profile"
2021-06-16 14:17:02 +02:00
86b916b008
enable optimization in every profile
2021-06-09 10:26:57 +02:00
e0c327bae2
Update Cargo.toml
...
Co-authored-by: Clément Renault <clement@meilisearch.com>
2021-06-08 11:39:10 +02:00
c82a382b0b
compile every build.rs with optimization
2021-06-08 11:19:22 +02:00
06c414a753
move the benchmarks to another crate so we can download the datasets automatically without adding overhead to the build of milli
2021-06-02 11:11:50 +02:00
9423310816
Introduce an helpers crate that export the database to stdout
2021-03-01 19:55:04 +01:00
a9a9ed6318
create workspace with meilisearch-error
2021-03-01 14:41:55 +01:00
79708aeb67
add milli as git dep
2021-03-01 14:41:20 +01:00
5ba58c1e9c
add Marin to authors
2021-02-28 10:09:56 +01:00
91d6e90d5d
enable faceted searches
2021-02-16 19:20:39 +01:00
fecf3d6fc1
Move the command lines helpers into different crates
2021-02-14 18:55:15 +01:00
e8639517da
Change the project to become a workspace with milli as a default-member
2021-02-12 16:15:09 +01:00
f8f02af23e
incorporate review changes
2021-02-04 13:21:15 +01:00
9af0a08122
post review fixes
2021-02-02 17:34:06 +01:00
17c463ca61
remove unused deps
2021-02-01 13:32:21 +01:00
6c63ee6798
implement list all indexes
2021-01-28 18:32:24 +01:00
74410d8c6b
architecture rework
2021-01-28 14:12:34 +01:00
433ac8c38a
Remove the ordered-float serde feature
2021-01-27 14:11:10 +01:00
61dbcfa44a
Bump the roaring to 0.6.4
2021-01-26 14:38:43 +01:00
51a37de885
Introduce the FacetValue enum type
2021-01-26 14:09:09 +01:00
6a3f625e11
WIP: refactor IndexController
...
change the architecture of the index controller to allow it to own an
index store.
2021-01-16 15:09:48 +01:00
686f987180
fix compile errors
2021-01-14 11:27:07 +01:00
ddd7789713
WIP: IndexController
2021-01-13 17:50:36 +01:00
1ae761311e
integrate with meilisearch tokenizer
2021-01-07 16:14:27 +01:00
0cd9e62fc6
search first iteration
2020-12-24 12:58:34 +01:00
02ef1d41d7
route document add json
2020-12-23 16:12:37 +01:00
1a38bfd31f
data add documents
2020-12-23 13:52:28 +01:00
55e1552957
update queue refactor, first iteration
2020-12-22 17:13:50 +01:00
7c9eaaeadb
clean code, and fix errors
2020-12-22 14:02:41 +01:00
77e951e933
Use the byte-unit crate to ease library usage
2020-12-20 12:00:37 +01:00
8c0ab106c7
initial commit
2020-12-12 13:32:06 +01:00
e7f2ab9138
Bump grenad to fix an indexing bug
2020-12-05 16:39:15 +01:00
0959e1501f
Introduce the FacetRevRange Iterator struct
2020-12-04 12:02:23 +01:00
61b383f422
Introduce the criteria update setting
2020-12-04 12:02:22 +01:00
a0adfb5e8e
Introduce a real pest parser and support every facet filter conditions
2020-11-23 16:43:55 +01:00
07a0c82790
Bump heed to 0.10.4 to use be able to lazily decode roaring bitmaps
2020-11-23 16:43:53 +01:00
38c76754ef
Make the facet level search system generic on f64 and i64
2020-11-23 16:43:52 +01:00
b255be93fa
Bump heed to 0.10.3
2020-11-23 16:43:49 +01:00
a18d9a1f87
Parse and store the faceted fields
2020-11-13 16:13:51 +01:00
640c7d748a
Modify the highlight function to support any JSON type
2020-11-05 13:59:32 +01:00
0408c9d66a
Move the http server into its own sub-module
2020-11-05 11:16:39 +01:00
4fded5bd0e
Bump heed to be able to reference a RoTxn from multiple threads
2020-11-02 12:49:23 +01:00
f0d028d3a4
Update the Transform struct to support JSON updates
2020-10-31 20:52:49 +01:00
9d47ee52b4
Generate a uuid v4 based document id when missing
2020-10-31 15:11:06 +01:00
085d3b9d94
Update heed to 0.10.0
2020-10-30 11:42:00 +01:00
b5d52b6b45
Prefer using a smallstr instead of a real String to reduce allocations
2020-10-29 14:32:32 +01:00
98fc24cbdf
Bump heed to fix a prefix iter bug
2020-10-28 10:55:21 +01:00
b44b04d25b
Serialize the CSV record values as JSON strings
2020-10-24 14:43:46 +02:00
802e925fd7
Switch to a JSON protocol for the front page
2020-10-21 18:26:29 +02:00
2210818114
Introduce the obkv heed codec
2020-10-21 15:51:48 +02:00
f948a03be2
Optimise the merge functions to avoid allocations
2020-10-20 16:40:50 +02:00
cde8478388
Replace the panic in the merge function by actual errors
2020-10-20 16:19:07 +02:00
35c9a3c558
Brodacast the updates infos to every ws clients
2020-10-20 11:19:34 +02:00
871222aebd
Introduce some new routes to handle live indexing
2020-10-19 16:06:43 +02:00
65e32fecb1
Move the binaries into one with subcommands
2020-10-19 13:44:17 +02:00
eca49e3a03
Introduce a notification channel for the UpdateStore
2020-10-18 16:37:37 +02:00
83c1db8763
Introduce the UpdateStore
2020-10-18 15:26:57 +02:00
9021b2dba6
Introduce the enable-chunk-fusing flag
2020-10-14 18:44:59 +02:00
f980422c57
Move from oxidized-mtbl to grenad
2020-10-14 12:47:32 +02:00
4e9bd1fef5
Bump oxidized-mtbl
2020-10-07 14:23:22 +02:00
433d9bbc6e
Use CompressionType::from_str rather than a custom function
2020-10-06 13:50:34 +02:00
4b819457c9
Enable the strucopt/clap warp help feature
2020-10-06 13:06:22 +02:00
770f29fd05
Bump the oxidized-mtbl dependency
2020-10-04 17:04:33 +02:00
acd2a63879
Introduce a simple FST based chinese word segmenter
2020-10-04 17:04:33 +02:00
68f4af7d2e
Improve the display of the number of processed documents
2020-09-29 16:08:58 +02:00
ed05999f63
Replace the arc cache by a simple linked hash map
2020-09-23 14:50:52 +02:00
Tamo