Commit graph

327 commits

Author SHA1 Message Date
Oscar Beaumont 3c440783b3
Spacedrop (#761)
* A hint of file drop

* backport from #671

* accept/reject Spacedrop working

* final cleanup

* Rename spacedrop.tsx to Spacedrop.tsx

* Update index.tsx
2023-04-27 04:41:33 +00:00
Brendan Allan 58e0542d8e
Give file paths pub_id and use it primarily (#754)
* give file paths pub_id and use it primarily

* fix object indexing

* file path location + sync

* remove future plan

* use id instead of pub_id for more indexer queries

* use chain_optional_iter

---------

Co-authored-by: Oscar Beaumont <oscar@otbeaumont.me>
2023-04-25 16:53:36 +00:00
Oscar Beaumont 9c62df9947 fix type exporting 2023-04-25 23:51:48 +08:00
Oscar Beaumont a4b84d31cc fix desktop 2023-04-25 22:47:41 +08:00
Oscar Beaumont fed3f2211f
Rspc v3 syntax (#758)
just fixed a couple of lines
2023-04-25 17:12:42 +08:00
Oscar Beaumont 85d0ddf6d3
rspc 0.1.3 prep (#756)
* Upgrade rspc + Specta + PCR

* fix jsonrpc executor

* fix invalidate excessive logging

* fix `Protected` type

* fix mobile

* clippy

* lol
2023-04-25 15:14:52 +08:00
jake 37b8d64642
[ENG-490] Add support for late key manager setup (#740)
* support for late key manager setup and remove password from onboarding

* mobile

* delete master-password.tsx

* remove secure temp keystore

* remove more km_config artifacts

* fix builds

* Fix most of the UI, there still some minor bugs

* Workaround portalled closing Popovers

* Restore search in core/src/api/mod.rs

* Fix Password eye icon closing Popover on click

* Fix keymanager setup submit button spinner not centered
 - Fix keymanager setup missing logic for disabling form
 - Adjust keymanager tab roundness to match app style

* Restore .gitignore

* Restore .prettierignore

* fixerino tests

---------

Co-authored-by: Utku Bakir <74243531+utkubakir@users.noreply.github.com>
Co-authored-by: Vítor Vasconcellos <vasconcellos.dev@gmail.com>
2023-04-22 09:10:21 +00:00
Ericson "Fogo" Soares fa5671f614
[Eng 495] Better job report fields (#726)
* Changing action column on job table

* Introducing action and is_background info

* Requested changes
2023-04-22 06:17:50 +00:00
Vítor Vasconcellos 24a3aedd7b
[ENG-384] Indexer rules editor (#723)
* WIP Create Indexer Rule UI + Backend api fix

* Complete IndexerEditor UI for creating new rules
- WIP testing to ensure all rules are parsed correctly
- Add utilities to satisfy the TypeScript typechecker
- Introduce a utility function to facilitate extracting information from an RSPCError
- Modify AddLocationDialog to utilize the aforementioned utility function

* Validation and submit logic is now functional
- Reorganize UI for improved user flow
- Implement validation and error messaging, replace some bare inputs with form inputs, and fix styling issues
- Resolve issues with post-processing of rules during submission
- Wrap editable portion of component in a `FormProvider` due to the `Form` being in a `Portal`
- Add specialized `ErrorMessage` component to `@sd/ui` library
- Update `AddLocationDialog` and `FormField` to utilize the new `ErrorMessage` component

* Fix submit not waiting for confirmation from backend
 - Reset form after submit
 - Disable form while submiting
 - Update form related dependencies

* Implement deleting an indexer rule
 - Modify indexer rule api route to disallow deleting default rules

* Fix form reset on error
 - Minor style adjustments
2023-04-22 04:30:11 +00:00
jake 4315cee147
[ENG-516] Prevent copies/cuts to the same source/dest (#748)
* add basic UI protection to stop cut/copies to the same path

* add rust src/dest protections

* fix pasting logic

* fix path canonicalization

* skip paths instead of overwriting them

* Using non-blocking ops and fixing some warnings

---------

Co-authored-by: Ericson Soares <ericson.ds999@gmail.com>
2023-04-22 02:11:59 +00:00
ameer2468 ee531a81fb
[ENG-473] - Job Manager UI Improvements (#745)
* add support for deleting a specific job

* wip

* Update JobManager.tsx

* Job manager design tweaks + popover open state ignore prop

* tweaks

---------

Co-authored-by: jake <77554505+brxken128@users.noreply.github.com>
2023-04-21 17:08:32 -07:00
Brendan Allan 469a3831dc
Explorer search (#738)
* basic file path search

* cleanup

* render explorer in search page

* update materialized path docs
2023-04-21 14:49:46 +00:00
nikec a97145348e
[ENG-295] Explorer media view (#730)
* remove unused onScroll

* Add open media file with double click

* Hide file thumb in inspector

* Enable media layout

* Add media layout options

* Add media view

* Fix vertical images

* Fix vertical square aspect

* query

* Update Location.tsx

* Some optimizations on explorer data query

---------

Co-authored-by: Ericson Soares <ericson.ds999@gmail.com>
2023-04-21 02:15:22 +00:00
Oscar Beaumont ce6d2d2fda
Eng 494 adding a location does not update the UI (#728)
* `sd_init.json` support

* bruh

* Add `sd_init.json` to developer docs

* Ran cargo clippy --fix to fix warnings

* Dafuq, cargo clippy --fix messed up cargo fmt

---------

Co-authored-by: Ericson Soares <ericson.ds999@gmail.com>
2023-04-20 22:05:57 +00:00
Ericson "Fogo" Soares f684660a90
Checking if new location is nested or already exists (#716)
* Checking if new location is nested or already exists

* Also checking for children locations on check nested

* Rust fmt
2023-04-19 05:03:28 +00:00
jake 288f4947e1
remove int-enum (#721) 2023-04-19 01:33:32 +00:00
Ericson "Fogo" Soares 7a792de090
[Eng 493] Job manager status showing incorrect (#722)
* Fixing path report for indexer on job manager

* Fixing warnings on migrator tests
2023-04-19 01:05:30 +00:00
Utku 7612458d76
Prisma Schema Comments (#720)
* add comments to prisma schema

* fix

* Update schema.prisma
2023-04-18 15:49:34 +00:00
Oscar Beaumont daadbf592a
Eng 488 fix app migrations system (#719)
* improve app level migration system

* basic unit tests for app migrator
We kinda don't want this going wrong so extra barrier is good.

* fix migrator tests
2023-04-18 08:51:01 +00:00
Vítor Vasconcellos 80fe8f98f1
[ENG-286] Set default indexer rules for location (#690)
* Implement multiple Glob matches in the same indexer rule
 - Replace matching logic to use GlobSet instead of simple Glob
 - Add `No OS protected` indexer rule to avoid indexing OS protected files
 - Enable `No OS protected` indexer rule by default

* Rust fmt

* Reduce `No OS protected` rule Glob list by using OR matches

* Add globs for android and ios files in `No OS protected` rule

* Add some more unix special path to be ignored

* Add a new property to IndexerRule to enable it by default when adding a new location:
- Modify the Prisma schema to add the default property to the IndexerRule model.
- Adjust the IndexerRule struct's create and save logic to consider the new property.
- Adjust AddLocationDialog to properly load the default indexer rules from the backend.
- Minorly improve IndexerRuleEditor to avoid adding duplicate entries to its assigned form field.
- Add editorconfig entries for SQL and Prisma types.

* rust fmt

* Add Windows users special folders and files to `No OS protected` rule

* Construct `No OS protected` globs from string vec
 - Don't repeat windows globs that require both a root and C: version

* Apply review feedback and add error handling for `IndexerRuleEditor`
- Revert manual changes made to init migration
- Create external migration that adds a `default` prop in `IndexerRule` model
- Remove redundant `useMemo`
- Replace destructuring of `useQuery`
- Provide feedback to user when a error occurs while querying indexer rules

* useMemo only for `indexerRulesIds` and not the whole `defaultValues` object
 - Remove logic to accpet unix root path for windows indexer `No OS protected` rule
2023-04-18 06:25:57 +00:00
Oscar Beaumont 8ea6a1c329
Automatic job progress (#717)
Remove time tracker task + sender level debouncing
2023-04-18 06:24:06 +00:00
Vítor Vasconcellos 3fbc2b909c
[ENG-461] Location creation fails for '/' (root) and 'C:' ([A-Z] drives) (#710)
* Fix issue with root locations failing to be added
 - Add `normpath` dependency to the core
 - Improve automatic name resolution for new locations

* Add `MetadataExt` trait to extend the `created` and `modified` methods of `Metadata`
- Prevent failure in the Indexer job caused by paths without associated created or modified times; assume current time for both when unavailable
- Rename possible location in root path from `/` to `Root`

* Improve hadling of location path on Windows
 - Normalize location's path to ensure a valid format on Windows
 - Improve error handling when adding a path
 - Avoid consuming symlinks on Linux by not normalizing location's path
 - Improve location naming logic
2023-04-18 06:12:02 +00:00
Brendan Allan 98d846827d
job parent id fix (#714) 2023-04-17 10:26:27 -07:00
Oscar Beaumont 2dd3d6f921
Job system improvements v2 (#707)
* Add `JobInitData` trait + change `StatefulJob::name` to a constant

* `StatefulJob::new`

* Move `invalidate_query` into `StatefulJob::finalize`

* Cleanup `spawn_job` signature

* Remove `queue_job` and run jobs from the `StatefulJob::finalize` method

* `StatefulJob::queue_jobs`

* `JobManager::ingest` return `Result`

* Remove `jobs.isRunning`

* Invalidation system direct push + batching invalidate

* Look ma, only a single clippy warning!!!!

* Error handling for JobManager

* Rust fmt

* Introducing Job hierarchy to enable job enqueuing

* Rust fmt again 🙄

* core.ts

---------

Co-authored-by: Ericson Soares <ericson.ds999@gmail.com>
2023-04-15 14:43:04 +08:00
Brendan Allan 8a525f6cc8 Revert "Give FilePath a pub_id (#691)"
This reverts commit 3c494ea96e.
2023-04-14 10:50:37 +08:00
jake 39e92cc7e7
update to ffmpeg 6.0.0 (#697) 2023-04-13 14:10:23 +00:00
Brendan Allan 3c494ea96e
Give FilePath a pub_id (#691)
* checkpoint - indexer parent ids need fixing

* use pub_id for file paths

* Removing LastFilePathIdManager and fixing some loose ends

---------

Co-authored-by: Ericson "Fogo" Soares <ericson.ds999@gmail.com>
2023-04-12 21:58:23 -07:00
Ericson "Fogo" Soares 1e05226efb
ENG-462 Object size is 0 when it shouldn't be (#702)
* Fixing file size not being collected from system

* Rust fmt
2023-04-13 01:06:27 +00:00
Vítor Vasconcellos 8224c7f146
ENG-460 Location creation fails for some writable directories (#696)
Remove read-only check when creating a location
- Revert location creation if writing the `.spacedrive` file or updating the library location manager fails
2023-04-12 12:51:32 +00:00
nikec 04c8a96943
[Desktop] Renamable files (#675)
* Add rename file option

* make Ericson's changes

* Prevent arrow selecting when renaming, start renaming with Enter option

* fix rename

---------

Co-authored-by: Jamie Pine <32987599+jamiepine@users.noreply.github.com>
Co-authored-by: Jamie Pine <ijamespine@me.com>
2023-04-11 20:43:19 -07:00
Oscar Beaumont 3d6afe5447
[ENG-414] P2P Shutdown (#688)
* `Event::Shutdown` in P2P system

* Mdns shutdown broadcast + await shutdown

* Fix mdns service name & prevent poll after shutdown
2023-04-11 15:05:50 +00:00
Oscar Beaumont 4c0e341e55
Add MetadataManager into sd-p2p (#686)
* Add `MetadataManager` into sd-p2p

* Add `update_metadata` helper to P2P manager

* Praise Clippy our lord and saviour

* change mdns advertisement log level

* Casually broke Linux
2023-04-11 10:20:14 +08:00
Vítor Vasconcellos 3304e8f6ce
QuickPreview Component (Needs test on MacOS) (#665)
* Add QuickPreview Component
 - Improve the handling of Range requests
 - Implement logic to answer HEAD and OPTIONS methods
 - Handle CORS pre-flight requests
 - Expand accepted file types
 - Improve error handling of invalid Range requests

* Fix linter errors
 - Add `use std::cmp::min` to custom_uri (Required on MacOS & Windows)
 - Improve logic for retrieving file information in QuickPreview.tsx

* More linter errors

* Simplify `QuickPreview` by extracting the logic for choosing the file preview tag to a `FilePreview` component
 - Fix the typo in `QuickPreview` props name
 - Remove the unused `handleMedia` ref
 - Move the remaining `QuickPreview` logic to the `transitions` callback
 - Simplify the `cors` return type in `custom_uri.rs`

* Refactor range handling in `handle_file` function
 - Move range handling logic to the initialization of the `range` variable
 - Replace `if let` with `match` to reduce code duplication
 - Don't export FilePreview
 - Export QuickPreviewProps

* Fix typo in `RangeNotSatisfiable` error message
 - Remove redundant variables

* Fixing cas_id generation on watcher
Some improvements on watcher file creation

* Rust fmt

---------

Co-authored-by: Ericson Soares <ericson.ds999@gmail.com>
Co-authored-by: Jamie Pine <ijamespine@me.com>
2023-04-05 21:15:13 -07:00
Brendan Allan 071af22b42
fix syncing (#679) 2023-04-05 19:47:10 -07:00
Vítor Vasconcellos b711fe8b27
Fix cas_id sample hashing logic (#672) 2023-04-04 13:59:19 +00:00
Brendan Allan ce9be10cdb fix core 2023-04-04 15:12:09 +08:00
Brendan Allan d44a70562b
some more sync impls (#669)
a lil more sync impls
2023-04-03 22:45:00 -07:00
Brendan Allan 5782af36eb
fix ts (#661)
* fix ts

* eslint
2023-03-31 00:22:54 -07:00
Ericson "Fogo" Soares b346e7ac52
Eng 372 location awareness improvements (#612)
* Complying with a pedantic Clippy

* Solving duplicated directories events
When creating a directory through MacOS's Finder, for some reason
FSEvents receives 2 Create Folder events that we have to handle

* Handling moving to trash bin and restoring on Mac
Still missing the feature to restore a directory and its children

* Now handling creation of empty files on MacOS

* Enabling restore of directories and its children

* Now working: moving a directory to another
inside the same location

* Now Indexer also remove file_paths not on fs

* Enabling multiple file moves on location watcher

* Fix Windows conditional compilation issues

* Fixing cas_id generation and bumping some deps

* Many Windows specific improvs and some refactors

* Rust fmt

* Using conditional compilation on extract inode function

* Linux fixes and some MaterializedPath improvements

* Rust fmt again

* Introducing tick behavior on location watchers

* Making LastFilePathIdManager more atomic

* Some vscode launch configs for lldb debugger

* Simplifying some lifetimes

* Making all watchers more consistent
2023-03-31 04:59:33 +00:00
Brendan Allan 21bacba28f
Sync debugging (#643)
* sync no networkey

* sync technically wired up

* fix p2p endianness for sync payload

* some sync debug stuff

* feature gate message production

* bump lockfile

* execute queries when sync messages disabled

* maybe fix ci?

---------

Co-authored-by: Oscar Beaumont <oscar@otbeaumont.me>
2023-03-29 00:38:48 -07:00
Vítor Vasconcellos 20f0e7f15a
Fix adding Locations that already have a .spacedrive file (#627)
* Handle RPC errors when adding Locations
 - Add custom error codes for NeedRelink and AddLibraryToMetadata errors in backend
 - Handle errors in AddLocationDialog form
 - Show custom error message for above errors and ask user to confirm action
 - Handle user action confirmation with appropriate RPC method (`locations.relink`/`locations.addLibrary`)

* Abstract openDirectoryPickerDialog logic to a function
 - Add 'Unkown Error' when code doesn't match a know one

* Fix bugs in logic to handle `locations.create` errors
 - Fix minor issue: AddLocationButton opened the dialog in desktop even when the user cancelled the file picker
 - Replace ternary with object for custom form error messages to allow for easier expansion
 - Break onSubmit logic up into functions
 - Remove incorrect `locations.indexer_rules.delete` call
 - Fix Checkbox element state not updating on external form state changes

* Adjust AddLocationDialog style to match other Dialogs in the app
 - Reset exception code at the end of `confirmAfterError`

* Rework how backend errors are handled when adding Locations
 - Change NeedRelink and AddLibraryToMetadata errors to use 409 (Conflict) status code and a static message.
 - Only enable custom submit logic in AddLocationDialog for 409 (Conflict) errors that have a known custom error form message.
 - Differentiate previous errors based on their messages instead of status code in AddLocationDialog onSubmit.
 - Add @hookform/error-message dependency for improved form validation error rendering.

* Invalidate library.list to ensure it is shown on sidebar

* Improvements to AddLocationDialog
 - Remove uncessary scope block and make useEffect depend directly on `form`
 - Move `form.handleSubmit` from `onSubmit` call to `handleSubmit` definition
 - Change basic Checkbox to RadixCheckbox and adapt onChange logic to onCheckedChange
 - Some simple styling improvements to the error message
2023-03-28 07:09:52 +00:00
Oscar Beaumont c2666f7f63
Sync over p2p (#606)
* sync no networkey

* sync technically wired up

* fix p2p endianness for sync payload

* some sync debug stuff

* feature gate message production

* bump lockfile

* execute queries when sync messages disabled

---------

Co-authored-by: Brendan Allan <brendonovich@outlook.com>
2023-03-21 06:03:08 -07:00
Jamie Pine 3ddd83000e
[Refactor] CAS id generation code simplify
Credit: ChatGPT 4
"In this simplified version, the code calculates the sample interval based on the file size and then uses a single loop to handle both the small and large file cases. The last sample is always taken from the end of the file."
2023-03-17 20:11:08 -07:00
Utku 902a3b5ba1
Onboarding, Spacedrop & Location Settings Screen & Styled API (#596)
* fix wrong current lib logic

* add delete lib dialog to LibraryGeneralSettings

* add delete lib to mobile LibraryGeneralSettings too

* onboarding screens

* move zxcvbn to @sd/client

* get started screen and bloom

* merge fix

* move generatePassword back to interface

* add useZodForm to mobile and match react-hook-form versions

* new lib screen

* Implement styled api

* create lib screen and some tweaks

* password input

* fix password meter comp

* new library style tweaks

* Fix remove password bug (interface)

* master password screen

* privacy screen

* creating lib screen

* hexagons are cool

* Expo 48

* keyboard handling

* fix P2P on IOS

* fix types

* asset script

* new icons

* Spacedrop screen

* Fix mobile asset imports

* fix import cycle warning

* Edit Location Settings screen and style changes on other setting screens

* fix library creating bug? hopefully lol

* move PasswordMeter to interface

---------

Co-authored-by: Oscar Beaumont <oscar@otbeaumont.me>
2023-03-17 02:00:02 +00:00
Jamie Pine 3b4248c836
[ENG-441] Explorer size toggle (#611)
explorer size toggle + light theme icons start + radix checkbox
2023-03-16 18:59:06 -07:00
jake 171f19bc71
[ENG-429] Crypto organization/refactor (#607)
* update comment

* key manager feature gating

* update deps & more feature gating

* fix keyring feature gating

* add an `exhaustive_read` function for crypto/stream

* restructure STREAM module

* move tests+shared utils

* clean up stream decryption

* further cleanup

* impl to `GenericArray` for `Nonce`

* update examples

* update refs & `use`s

* fix `Nonce` -> `GenericArray` conversions

* better `Protected` conversions + remove `Password` type

* a work of art

* finishing touches

* some API changes

* rename `StreamX` to `X`

* fix everything else

* separate `primitives` from `types`

* update imports & fix build
2023-03-15 16:09:36 +00:00
Oscar Beaumont 1978e2ff48
Eng 432 refactor mdns service/state (#604)
clean up mdns state and refactor advertisement system
2023-03-13 18:05:46 +08:00
Oscar Beaumont 0c25239c53
Typesafe stream handling (#603)
* typesafe stream handling
Now it's impossible for your to respond to a broadcast stream. Previously this would just fail due to the TCP connection having been closed by the sender.

* remove connected_peers state from `SpaceTime`

* use `OutEvent` instead of `tokio::spawn` for `SpaceTime` events

* Polling for events > tokio::mpsc

* extend max length of sync messages to ~4GB + broadcast done message

* Fix "actOS" operating system decode error
2023-03-13 16:58:35 +08:00
Ericson "Fogo" Soares 4a6b057872
[ENG-371] Subpath shallow re indexing (#599)
* Introducing sub path indexing for IndexerJob

* Introducing shallow indexer job

* Some feature flags to avoid warnings at pnpm prep

* Many fixes at shallow indexer job

* Changing how we implement the bookkeeping for file_path ids
Now we're account for independent locations, and also integrating
better with the LibraryContext, instead of using a static global

* Making indexer job reentrant

* Introducing shallow file identifier job
And a bunch of minor refactors

* Rust fmt

* Removing light scan from getExplorerData query
Light scan is a mutation, so we can call it on useEffect function from
the Explorer component in the frontend, when a location_id or the explorer
path changes

* Handling job early finish on init for identifier

* Only invalidate query if we have orphan paths

* Introducing ShalowThumbnailerJob

* Clippy warnings about `into_iter()`

* Naming scheme for Prisma's selects and includes

* Invalidating getExplorerData at thumbnailer

* Small mistakes lol

* Some nitpicks with pnpm prep

* Rust fmt

* Changing indexer's walk log to `trace!`

* Not deleting all file_paths on location fullRescan

* TS typecheck

* Removing `file_path` selection just with id
2023-03-10 16:25:58 +00:00
Oscar Beaumont 7c5f760fbd
Basic Spacedrop working with rspc! (#601) 2023-03-09 06:23:17 -08:00