dnrops
/
cryptpad
mirror of https://github.com/xwiki-labs/cryptpad
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
13,530 Commits 92 Branches 135 Tags 578 MiB
Commit Graph

495 Commits

Author SHA1 Message Date
ansuz 466072d03b read files starting from the oldest known point of relevance when computing indices 2020-11-26 12:15:02 +05:30
yflory 93ddf51564 Merge branch 'download' into cacheRT 2020-11-24 16:46:22 +01:00
ansuz cd77b5ffc8 Merge branch 'soon' into staging 2020-11-19 13:18:08 +05:30
ansuz faa84bcbd7 describe arguments for 'completeUpload' RPC 2020-11-19 12:16:02 +05:30
yflory bd4b44476d Archive and restore UI in the admin panel 2020-11-18 16:49:40 +01:00
ansuz e185008846 complete uploads in child processes 
also fix a nasty race condition for unowned file uploads
 2020-11-18 18:26:21 +05:30
ansuz b155f7b291 increase child-process timeout and improve error handling 2020-11-17 16:09:47 +05:30
David Benqué ead97d8f41 Remove FAQ and references to it 2020-11-16 11:06:48 +00:00
yflory 5946b10d0b No page reload when the cache is corrupted onReady 2020-11-06 15:00:58 +01:00
ansuz 1f1b40b83d move some implictly global state to env.js 2020-11-03 16:48:31 +05:30
ansuz aa547a7b76 align nodejs http headers with example nginx 2020-11-03 16:48:31 +05:30
ansuz 5cf8ca70e6 don't pin falsey document ids 2020-11-03 16:48:31 +05:30
yflory c9a3cad78b Fix various issues with restricted pads 2020-11-03 16:48:31 +05:30
yflory d16cc2472d First prototype of an offline cache 2020-11-03 10:49:13 +01:00
David Benqué 66d434b583 Remove about page from defaults and add XXX for future pages 2020-10-30 14:29:15 +00:00
ansuz 034472d3dd move some implictly global state to env.js 2020-10-27 12:47:15 +05:30
ansuz 0bc2433e02 Merge branch 'accessBug' into staging 2020-10-27 09:09:09 +05:30
ansuz d95b0954f9 Merge branch 'fix-printing' into staging 2020-10-27 08:13:58 +05:30
ansuz a2b79d84b8 align nodejs http headers with example nginx 2020-10-27 08:12:23 +05:30
ansuz f7bd3bdc23 don't pin falsey document ids 2020-10-26 17:34:41 +05:30
yflory 69c26fe8c7 Fix various issues with restricted pads 2020-10-23 15:44:47 +02:00
ansuz d72e053560 make a note to improve stream timeout error handling 2020-10-22 11:24:58 +05:30
ansuz 67430de7ff Make efforts to avoid closing streams mid-read 
1. Close streams when you're done with them
2. Close streams if they seem to have been abandoned
 2020-10-22 11:17:03 +05:30
ansuz fbfb25bf29 lint compliance 2020-10-21 21:29:52 +05:30
ansuz 100b417646 guard against several serverside typeErrors 
and warn in cases where they would have occurred
 2020-10-21 18:24:54 +05:30
ansuz e8428a2a73 prevent a case of multiple callbacks 2020-10-21 18:13:10 +05:30
ansuz fbc9edd795 rename latest admin commands and implement metadata getter/remover 2020-10-20 17:12:26 +05:30
ansuz 488ec93ece allow expert admins to get and clear cached channel indices 2020-10-20 17:07:55 +05:30
ansuz faa7ebf399 Fix some bugs with integrated eviction 
1. implement SET_LAST_EVICTION as an admin command, not a decree
2. expect a return value from Env.evictInactive and expose it via Env.evictionReport
 2020-10-15 13:15:39 +05:30
ansuz ac322c8e82 make the eviction script call back with accumulated statistics 2020-10-14 17:23:53 +05:30
ansuz a29c5641b8 introduce some artificial delays in the eviction script 
...and reduce the margin of error in its bloom filters
 2020-10-14 17:22:31 +05:30
ansuz 90f046f896 prevent long-running worker tasks from timing out 2020-10-14 17:18:57 +05:30
ansuz 6e57366b7f address a file descriptor leak 2020-10-13 13:14:39 +05:30
ansuz a2c0d2165b implement a few more admin decrees 2020-10-13 11:36:30 +05:30
ansuz abd84665ae lint compliance 2020-10-13 11:00:45 +05:30
ansuz 6ec5171518 add support for changing a few more Env parameters at runtime 2020-10-13 10:56:40 +05:30
ansuz adb988058d stop logging dev/fresh mode 2020-10-12 19:19:57 +05:30
ansuz f2ec9cbe33 load premium and customLimits to avoid evicting them even if inactive 2020-10-12 19:18:12 +05:30
ansuz 45b063e378 suppress irrelevant errors from Decrees.load 2020-10-12 19:15:48 +05:30
ansuz 854e4c06ff report lastEviction in an admin INSTANCE_STATUS 2020-10-12 17:52:15 +05:30
ansuz 471a9a33ac take note of a minor error with pin RPCs 2020-10-12 17:51:38 +05:30
ansuz f86409e1ba take note of which Environment variables should be made mutable via decree 2020-10-12 17:50:40 +05:30
ansuz 6d13a785c6 downgrade a non-critical XXX to a FIXME 2020-10-12 17:42:41 +05:30
ansuz 46ebd7b40b remove unsupported storage configuration parameters 2020-10-12 17:40:21 +05:30
ansuz 369c92c01d initialize Env from server and deduplicate several attributes 2020-10-12 17:39:53 +05:30
ansuz b5d3a10dc2 guard against possible typeErrors from unvalidated config input 2020-10-09 13:28:13 +05:30
ansuz 964aa2bb79 validate custom limits provided via config at launch 2020-10-08 19:07:00 +05:30
ansuz f4f803ccd9 reuse Keys.canonicalize to accept any pubkey format 2020-10-08 19:07:00 +05:30
ansuz 914139dbfa lint compliance 2020-10-08 19:07:00 +05:30
yflory 51e1b9b043 Fix custom limit not applied when using a raw edPublic 2020-10-08 19:07:00 +05:30
ansuz 145386e944 suppress an inconsequential error 2020-10-08 19:07:00 +05:30
ansuz aa35e092f7 update an inaccurate comment 2020-10-08 19:07:00 +05:30
ansuz 928b7e3583 implement GET_LIMITS admin rpc 
...and label stored usage limits according to their source
 2020-10-08 19:07:00 +05:30
ansuz 5029281a53 implement RM_QUOTA decree 2020-10-08 19:07:00 +05:30
ansuz af47b2cab1 implement ADD_QUOTA admin decree 2020-10-08 19:07:00 +05:30
ansuz 998f2267c4 ensure the path to the decree file exists before attempting to write to it 2020-10-08 19:07:00 +05:30
ansuz c2918ac888 fix inverted conditional 2020-10-08 19:07:00 +05:30
ansuz b62d4387a6 drop support for SET_DEFAULT_STORAGE_LIMIT 
it's redundant now that we support persistent admin decrees
and it was never exposed via the UI anyway
 2020-10-08 19:07:00 +05:30
ansuz 0c5db31fbc refactor quota logic for external usage 2020-10-08 19:07:00 +05:30
ansuz bd6234c5bc lint compliance and some notes 2020-10-08 19:07:00 +05:30
ansuz 1781ee2585 implement 'UPDATE_DEFAULT_STORAGE' as a decree 2020-10-08 19:07:00 +05:30
ansuz 7aa7d5978f implement persistence of config changes from admin panel 2020-10-08 19:07:00 +05:30
ansuz fcfcb3dca6 fix history trim for channels with access lists 2020-10-07 18:44:24 +05:30
ansuz d509fd8b8f downgrade severity of several XXX notes 2020-10-07 17:09:46 +05:30
ansuz 1ba080739b Merge branch 'X' into staging 2020-10-07 16:28:08 +05:30
ansuz cfcfe2f65f enable APIs for XLSX export in firefox 2020-10-07 14:47:43 +05:30
yflory e698241ee9 Merge branch 'historyOO' into staging 2020-10-05 16:41:23 +02:00
ansuz 04d1bc9cd7 archive inactive data in one of the server's workers 2020-09-30 14:45:52 +05:30
ansuz 6ada5fd751 always consider premium accounts to be active 2020-09-29 18:48:23 +05:30
ansuz 5358fab615 more WIP inactive account eviction 2020-09-29 18:04:51 +05:30
ansuz a9b95865ba Merge branch 'staging' into inactive-accounts 2020-09-28 17:56:30 +05:30
ansuz d9219f4ecc add a note for later 2020-09-25 14:27:22 +05:30
ansuz 5749b6fd7b add an admin RPC to archive documents 2020-09-25 14:24:26 +05:30
ansuz d794e0b48f include inactive accounts in the archival script 2020-09-24 19:02:22 +05:30
yflory 25606b92bf lint compliance 2020-09-22 15:27:16 +02:00
yflory 4f147d4fd2 Add support for onlyoffice history 2020-09-18 17:54:57 +02:00
yflory b31707098a Improve getHistoryRange to support 'toHash' 2020-09-18 12:42:09 +02:00
ansuz 95778cc7c9 respond with ENOENT instead of E_NO_OWNERS when archiving a file that does not exist 2020-07-23 09:30:04 -04:00
ansuz b04133e6c3 remove some dead code 2020-07-23 09:28:00 -04:00
ansuz a21c3372c0 avoid a potential type error 2020-07-23 09:27:29 -04:00
yflory 04af753a81 lint compliance 2020-07-01 12:10:12 +02:00
ansuz 03be102ce4 deduplicate some code by reusing the client definition on the server 2020-06-30 14:33:00 -04:00
ansuz ebcc9a069b add server-side support for the new format of public signing keys 2020-06-25 13:14:26 -04:00
ansuz 48deced859 fix serverside index corruption 2020-06-16 04:08:03 -04:00
ansuz 5b2929a6fc suppress some useless info from some commonly logged errors 2020-05-06 13:50:48 -04:00
ansuz 9f1f01f3b4 disable worker task queueing and address a probable memory leak 2020-05-05 18:06:28 -04:00
ansuz b0af6b5488 fix an incorrect condition when checking for available server workers 
Also, log when the RPC queue has a backlog and when it is drained
 2020-05-05 10:53:42 -04:00
ansuz 4037270a23 add a FIXME note for a sketchy implementation in history keeper 2020-05-04 15:46:08 -04:00
ansuz 02347d134a queue signature validation per-channel to ensure correct ordering 2020-04-30 18:18:05 -04:00
ansuz 9975c27e84 increase timeout from one minute to three and decrease the number of parallel jobs per worker 2020-04-29 17:24:38 -04:00
ansuz 488f876ad6 lint compliance 2020-04-29 17:23:29 -04:00
ansuz d35b015c3f log some info on successful file uploads 2020-04-28 17:12:03 -04:00
ansuz 37d3995ac1 unify format of console output with stored logs 2020-04-28 17:05:15 -04:00
ansuz 8a5d012edf add special error handling for a few special serverside errors 2020-04-28 16:57:52 -04:00
ansuz 8bee95d13d improve error reporting in GET_HISTORY 2020-04-28 16:56:11 -04:00
ansuz e05891184e fix regression in maxWorkers 2020-04-21 12:23:48 -04:00
ansuz b56c73be6e delegate more work from getOlderHistory to the worker 2020-04-17 17:01:47 -04:00
ansuz fd169ff39c close streams after 2 minutes instead of 5 
Hopefully this helps to even out our memory and CPU usage.
There's also some improved labels so that IF streams time out
we will know the exact origin of the offending call.
 2020-04-17 15:05:48 -04:00
ansuz bac8ef3c6e lint compliance 2020-04-17 11:43:54 -04:00
ansuz 4e57e390da write tasks inside of workers 2020-04-16 16:37:24 -04:00
ansuz ba6faca02e make the number of workers configurable 2020-04-16 13:53:45 -04:00
ansuz 0465f31a45 add a FIXME note 2020-04-16 13:02:27 -04:00
ansuz 04ab7f538a recover from worker faults and unify worker types 2020-04-16 13:02:07 -04:00
ansuz e0a6852b79 correctly serialize an error log 2020-04-16 10:45:18 -04:00
ansuz 9ed82640bf Merge branch 'worker-queue' into staging 2020-04-15 19:32:42 -04:00
ansuz e8b1fcf710 solve some cases where crashing workers could result in an invalid state for the queue 2020-04-15 13:59:54 -04:00
ansuz 9ed26cfeb0 lint compliance 2020-04-15 12:40:18 -04:00
ansuz dc0460780f Merge branch 'soon' into staging 2020-04-15 09:44:30 -04:00
ansuz 74567e0cf4 return process memory usage in GET CACHE STATS RPC 2020-04-14 15:53:58 -04:00
Martin Alexander Neumann e3f42cbc20 Allow config.defaultStorageLimit to be zero bytes 2020-04-11 20:46:09 +02:00
ansuz 89262cd29e resolve silly conflict and merge staging 2020-04-09 16:35:24 -04:00
ansuz e527867e2e queue blob and channel deletions per-user 2020-04-09 16:29:52 -04:00
ansuz bd3e18d1a8 only cache a user's pinned channels if there are no errors when loading them 2020-04-08 12:08:26 -04:00
ansuz bef18a9320 hotfix 2020-04-08 10:28:26 -04:00
ansuz 7fac997e93 increase some file storage timeouts related to streams 2020-04-08 09:38:48 -04:00
ansuz 95965c1dee keep a parallel implementation of the pin loader to validate the new one 2020-04-07 20:03:41 -04:00
ansuz a57bf94058 remove XXX note 2020-04-06 11:31:31 -04:00
ansuz c53675c9d5 WIP worker rpc call queue 2020-04-06 10:28:26 -04:00
ansuz d802173325 remove an invalid line 2020-04-06 10:18:11 -04:00
ansuz 2b3f6e3464 Merge branch 'soon' into staging 2020-04-03 10:49:51 -04:00
ansuz adfd1eaaa2 avoid a silly typeError in the event of an error 2020-04-02 22:57:57 -04:00
yflory 32406a5995 Merge branch 'master' into staging 2020-04-02 10:23:09 +02:00
ansuz 834c96a4fc update comments and remove some notes 2020-04-01 17:43:40 -04:00
ansuz ed5d97f899 WIP history-keeper fixes 2020-04-01 11:48:16 -04:00
ansuz eddbe80eef Merge branch 'soon' into staging 2020-03-30 18:30:34 -04:00
ansuz 3d8e78a268 fix overly broad config interpretation for daily check 2020-03-30 18:27:14 -04:00
ansuz e6ec891d9a improve admin rpc for diagnosing memory used by the cache 2020-03-30 18:26:04 -04:00
ansuz 77961e3954 trim lookup tables for channels without checkpoints 2020-03-30 18:03:37 -04:00
ansuz db9eaa8d4f Merge branch 'soon' into staging 2020-03-27 19:59:45 -04:00
ansuz 6d0dee979a allow admins to inspect index/metadata caches 2020-03-27 19:59:26 -04:00
ansuz cbd3547814 remove log statements 2020-03-27 17:20:57 -04:00
ansuz 9058a59555 reassign db tasks if the responsible worker fails 2020-03-27 17:17:42 -04:00
ansuz 172823c954 lint compliance 2020-03-27 16:59:41 -04:00
ansuz b5649707d1 export 'tryParse' command 2020-03-27 15:36:34 -04:00
ansuz 64b0879984 improve logging for parse errors 2020-03-27 15:15:54 -04:00
ansuz 3f86b6141e rename worker processes 2020-03-27 15:14:45 -04:00
ansuz d8a88cb4ca run expiration tasks in a worker instead of the main process 2020-03-27 14:38:58 -04:00
ansuz b0179eaad9 drop XXX note 2020-03-27 14:25:07 -04:00
ansuz 9dbd32758a improve worker logging and move blob deletion to worker processes as well 2020-03-27 13:38:27 -04:00
ansuz 5f69fc18d0 suppress some noisy errors 2020-03-26 17:11:43 -04:00
ansuz 5f2d7c8dcf increase worker rpc wait time before timeout 2020-03-26 15:53:00 -04:00
ansuz 33e8e65507 handle errors in the server's workers 2020-03-26 15:13:20 -04:00
ansuz a4c8039cc7 improve error handling with rpc response API 2020-03-26 14:45:24 -04:00
ansuz cb53bd1c15 lint compliance 2020-03-26 14:44:37 -04:00
yflory e15196bd53 Merge branch 'reconnect' into staging 2020-03-26 15:24:38 +01:00
ansuz 14feef1757 calculate pin list checksums in a worker 2020-03-25 18:37:42 -04:00
ansuz 70a0d4efb4 move more database reads into the database worker 2020-03-25 17:43:57 -04:00
ansuz bc13a21796 give fileStreams a little bit more time before closing them 2020-03-25 11:50:45 -04:00
ansuz 4ba36a9173 load user pins in the database worker 2020-03-25 11:39:14 -04:00
ansuz 50e8893b24 move the 'getOlderHistory' call into the database worker 2020-03-25 10:54:29 -04:00
ansuz 471e374533 compute metadata in the same child process that builds indexes 2020-03-24 17:43:15 -04:00
ansuz 479b76f848 lint compliance 2020-03-24 17:40:07 -04:00
ansuz c2fcba31e3 remove an XXX 2020-03-24 15:02:46 -04:00
ansuz 967ca6afa9 suppress ENOENT errors when checking empty chat channels 2020-03-24 15:02:26 -04:00
ansuz 40251948d4 check authenticated rpc signatures in separate threads 2020-03-24 14:31:40 -04:00
ansuz 0e09c73a60 Merge branch 'master' into soon 2020-03-24 13:32:42 -04:00
ansuz a2facc4a0d
Merge pull request #513 from 7adietri/patch-1 
Add shutdown() to non-disk logger
 2020-03-24 13:32:13 -04:00
ansuz e128683ffb call back with errors instead of throwing within the Storage module 2020-03-24 13:31:03 -04:00
ansuz 1240f4a2a5 Merge branch 'master' into soon 2020-03-24 13:04:25 -04:00
ansuz 90899aa2f4 start reviewing and removing XXX notes 2020-03-24 13:03:49 -04:00
yflory fe14399e67 Merge branch 'whiteboard' into soon 2020-03-24 16:15:13 +01:00
ansuz 2ef0af29e5 guard against index worker rpc responses for a txid with no supplied callback 2020-03-23 22:03:59 -04:00
ansuz ee8d5c9c36 fix the premiumUploadSize validation to check the correct attribute 2020-03-23 16:13:48 -04:00
yflory dc0b26da0b Merge branch 'soon' into staging 2020-03-23 12:33:17 +01:00
Alexander a190af079d
Add shutdown() to non-disk logger 
The script `evict-inactive.js` calls `Log.shutdown()` at the end, which currently fails if logging to disk has been disabled via `logPath`.
 2020-03-21 14:43:25 +01:00
ansuz ed63d98ab0 Merge branch 'scaling-index' into soon 2020-03-20 18:57:51 -04:00
ansuz 590e249500 resolve merge conflict 2020-03-20 13:07:35 -04:00
ansuz 31c7cecaf9 add a few checks for safety 2020-03-20 10:54:01 -04:00
ansuz cea9705bbe Merge branch 'aggressive-stream-closing' into staging 2020-03-20 10:50:17 -04:00
ansuz faa133aab8 remove an XXX 2020-03-20 10:22:48 -04:00
ansuz d386e223e4 simplify open/close of writeStreams 2020-03-19 17:33:22 -04:00
ansuz 32cd0f3c4d increase timeout value from 15s to 45. ought to be enough for anybody 2020-03-19 16:13:24 -04:00
ansuz 05a4e86cdb close streams whenever we finish using them. time out if necessary 2020-03-19 16:11:24 -04:00
ansuz 4522ffa18a compute indexes in child processes 2020-03-19 10:46:18 -04:00
ansuz f8ad649b45 [style] bail out early to avoid nesting 2020-03-18 10:30:42 -04:00
ansuz 019e5e708b wrap workers in a function scope and add a validateMessage method to HK's Env 2020-03-17 17:23:35 -04:00
ansuz 5467e1ffac replace ad-hoc response handler with Util.response 2020-03-17 16:52:41 -04:00
ansuz 9e85a1411e abstract the logic around worker choice out of message handling 2020-03-17 16:06:01 -04:00
ansuz 631ea54b49 lint compliance 2020-03-17 15:40:06 -04:00
ansuz 67dd4a608c Merge remote-tracking branch 'origin/scaling' into scaling2 2020-03-17 15:36:58 -04:00
ansuz b2fcde87d8 implement an admin command to update the default storage limit without a restart 2020-03-17 10:11:52 -04:00
yflory 923616aef0 Use each process in order instead of using a random one 2020-03-17 15:10:23 +01:00
ansuz f345998956 fix undefined reference 2020-03-17 10:10:20 -04:00
yflory fb0eb1b20c Use more subprocesses 2020-03-17 14:18:41 +01:00
yflory 0d636dabc9 Check signature for history keeper in a different process 2020-03-17 13:29:53 +01:00
ansuz 30f17040ac close streams when possible, simplify some code, enforce asynchrony 2020-03-16 16:13:38 -04:00
ansuz 2a9780abcf lint compliance 2020-03-16 12:29:26 -04:00
yflory 578042154e Better reconnect after invalid lastKnownHash 2020-03-13 17:39:16 +01:00
ansuz 2d47e7e2cf implement an admin rpc to fetch ulimit data 2020-03-12 12:09:44 -04:00
ansuz 81c7416095 lint compliance 2020-03-12 11:40:46 -04:00
ansuz f644c24362 add an admin-rpc to count the number of open files 2020-03-12 11:35:16 -04:00
ansuz 1230057970 disable IS_PAD_PINNED 2020-03-12 11:31:08 -04:00
ansuz e1a0daac9c possible server fixes 2020-03-12 11:23:43 -04:00
ansuz bc034d95a3 enforce asynchrony in 'gethistoryOffset' 2020-03-12 11:13:23 -04:00
ansuz d4038344ba stop iterating over tasks if you can't read the root directory 2020-03-11 16:34:00 -04:00
ansuz 4e8ebac3c1 guard against the possible non-existence of metadata for a channel 2020-03-09 17:42:10 -04:00
ansuz b32f3e34af fix local worker debugging on firefox 2020-03-06 11:43:34 -05:00
ansuz 6d8cdca7a3 add 'rejected' field to metadata when getMetadata requests are blocked 2020-03-05 12:09:18 -05:00
ansuz 9d8bb43d03 finally untangle metadata and index caches 2020-03-04 12:56:33 -05:00
ansuz 32d769447a begin standardizing our method of streaming lines from files 2020-03-04 11:38:14 -05:00