Commit graph

42 commits

Author SHA1 Message Date
Vítor Vasconcellos 2c777e53f1
Fix core test (#2386)
* Fix core test

* Import CompressedCRDTOperations

---------

Co-authored-by: Ericson "Fogo" Soares <ericson.ds999@gmail.com>
2024-04-24 23:37:38 +00:00
Oscar Beaumont 0b6bd050a0
Fix main (#2381)
* fix

* fix
2024-04-24 10:09:18 +00:00
Oscar Beaumont 643bd3a142
Block size (#2377)
Block size + some Clippy
2024-04-24 08:27:31 +00:00
Oscar Beaumont d9b31a90b3
Fix sd-server auth (#2331)
fix
2024-04-15 12:52:59 +00:00
Oscar Beaumont 9f5396133b
Basic HTTP auth for sd-server (#2314)
* basic http auth

* fix types

* Fix

* auth docs
2024-04-12 03:31:38 +00:00
Vítor Vasconcellos 08c42a36fe
[ENG-1624] E2E Tests (#2192)
* Basic cypress test

* Add e2e test command

* Add full e2e test for creating and deleting a library

* Add cypress to CI

* Fix cypress CI
 - Use arm macos for running cypress
 - Setup rust on cypress job
 - Increase cypress CI timeout

* Remove unused APPLE_SIGNING_IDENTITY envvar

* Update deps again + Try to fix Cypress CI setup

* Update deps again

* Only test default location if it exists

* Use latest v18 NodeJS in CI

* Increase minimum node version due to updated dependency
 - @typescript-eslint/eslint-plugin now requires node >=18.18

* Fix testing failing for optional default locations

* Enable video recording
 - Fix screenshot and video artifact uploading

* Fix location test
 - Use correct location name, instead of trying to derive it from the internal id
2024-03-19 20:22:17 +00:00
jake 4032677024
Fix localStorage wiping on every launch (#2021)
* fix: wait until libraries have loaded before checking the count

* fix: include new core version in lockfile

* fix: moved value

* fix: check the `libraries/` data dir manually

* Clear localstorage via localStoarge.clear() (#2040)

only clear localstorage in desktop using js

---------

Co-authored-by: Brendan Allan <brendonovich@outlook.com>
2024-02-02 15:55:20 +00:00
jake f9e827512f
[ENG-1540] Remove localStorage if no libraries are created (#1945)
* feat: clear localstorage on start if no libraries are found

* chore: add a check to see if the node is for desktop or server use

this will prevent desktop/app localstorage from being cleared if it's a server being hosted

* fix: add secondary localstorage directory for macos

for some reason the app/onboarding progress is kept in `Library/WebKit/Spacedrive` (at least in dev?) so it also needs clearing

* feat: delete cache directories also

* chore: iterate over paths

* feat: add support for windows `localStorage` by using `data_local_dir`

* docs/style: better comments and logs

* feat: remove both `AppData` local and roaming on Windows

* fix: mobile builds

* chore: tweak error message for windows
2024-01-29 12:32:48 +00:00
Brendan Allan 4962b1160b
[ENG-1488] Cloud sync message compression (#1922)
* CRDTOperationWithoutInstance

* almost there

* fully compress messages

* implement more of sd-cloud-api

* sd-cloud-api-ify

* landing -_-

* openssl

* clippy

* idk

* bruh

* wut

* ahhh right
2024-01-09 10:21:23 +00:00
Brendan Allan 5371e9f961
[ENG-1512] Replace sync operation types with single type (#1900)
* replace multiple op types with crdt_operation

* cleanup
2023-12-21 05:58:03 +00:00
jake c79c190c78
[ENG-1426] Apply clippy --fix with moderate lints (#1779)
* chore(clippy): fix

* chore: formatting

* fix: add cfg back to macos crate

* remove cfg to see if clippy works

* Restore macOS crate level conditional compilation

* More warns fixing

---------

Co-authored-by: Vítor Vasconcellos <vasconcellos.dev@gmail.com>
Co-authored-by: Ericson Fogo Soares <ericson.ds999@gmail.com>
2023-11-28 17:50:11 +00:00
Brendan Allan 3786e93a78
Remove client secret from apps (#1423)
* remove client secret from apps

* dumb
2023-10-03 14:45:19 +00:00
Brendan Allan 861cfb8b40
Use actual oauth clients + ship ids and secrets (#1415)
* use actual oauth clients + ship ids and secrets

* re-enable features

* use prod as default api url
2023-10-02 09:36:43 +00:00
Oscar Beaumont f821bb9a0f
Stop App Freezing (#1257)
* Drop Tauri custom URI handler
Me no likely but it has to be done.

* fix app startup with location

* fix "Add Location" button on web

* Serve correct content range

* Backport changes from 08ba4f91

* none of my homies like panics

* minor fixes

* fmt with new Rust version
2023-08-31 06:54:40 +00:00
Oscar Beaumont 7bf0c0ae4a
[ENG-994] Minor P2P fixes (#1238)
* mdns advertisement on detection

* fix bug in P2P discovery

* Add `P2PEvent::ExpirePeer`

* React properly

* Consistent tracing versions

* better logger config

* Reduce excessive logging from P2P

* Fix panic log hook

* Remove `dbg`

* Fix bug in thumbnailer remover

* wip: Connected peers in UI

* `Sync` is large pain

* init services after library load

* improve error handling of logger

* Sync protocol shutdown

* Add "updater" feature

* fix commands bindings

* nlm state debug query

* nlm debug status

* Make TS happy

* Add `RemoteIdentity` to libraries debug info

* Improve debug data presentation

* Among us level sus

* minor fix panic hook

* Fix EOF issue

* fix instance not being added on pairing

---------

Co-authored-by: Utku <74243531+utkubakir@users.noreply.github.com>
Co-authored-by: Ericson "Fogo" Soares <ericson.ds999@gmail.com>
2023-08-27 06:41:26 +00:00
Brendan Allan fd236a1b57
[ENG-679] Reserve ids for built in indexer rules (#909)
* indexer rules pub ids

* should work?

* better migrator

* errors

* debugging

* maybe?

* double migrate

* please

* maybe fix?

* update lockfile

* SD_ACCEPT_DATA_LOSS message

* put tracing back

* dumb

* fix system indexer rule ui
2023-06-06 14:42:52 +00:00
Oscar Beaumont 5f71e0c1f6
[ENG-546] Improve error handling (#802)
* log to disk

* remove some unwraps

* panicless

* some p2p error handling

* clippy moment

* Fix `<ErrorBoundary />`

* open logs button

* 39 to 0

* fix types

* update deps and comment out broken tests

* clippy

* more clippy

* upgrade rimraf - https://github.com/isaacs/rimraf/issues/259

* regen broken lockfile

* update `notify` and update `commands.ts`

* more clippy (pls work)

* allow deprecated for p2p (hopefully temporary)

* a

* upgrade deps for p2p

* do betterer

* do it correctly

* remove unused import

* improve core startup error + bc keypair

---------

Co-authored-by: Utku Bakir <74243531+utkubakir@users.noreply.github.com>
Co-authored-by: brxken128 <77554505+brxken128@users.noreply.github.com>
2023-05-30 09:24:05 +00:00
Oscar Beaumont a20f6262fd
[ENG-659] Spacedrive On Jamie's NAS (#866)
* serve dem

* 0% chance of working

* With a side of Vite plz

* no CI for now

* Clippy plz work

* undo

* bruh

* plz work
2023-05-29 10:47:59 +00:00
Oscar Beaumont c2666f7f63
Sync over p2p (#606)
* sync no networkey

* sync technically wired up

* fix p2p endianness for sync payload

* some sync debug stuff

* feature gate message production

* bump lockfile

* execute queries when sync messages disabled

---------

Co-authored-by: Brendan Allan <brendonovich@outlook.com>
2023-03-21 06:03:08 -07:00
Oscar Beaumont a9fceae819
improvements to spacedrive:// custom URI protocol (#550)
* fix `spacedrive://` custom protocol on Windows
(hopefully)

* custom protocol using `http::Response` + fix broken web

* import patches before App on web

* use `http::Request` for input to `handle_custom_uri`

* break  into dedicated file + error handling

* serving files via custom protocol

* cargo fmt because vscode did cringe

* lru cache to reduce video chunk request time

* add  helper to JS

* clippy be like

* remove duplicate Open buttons in context menu

* fix Linux 🙏

* no shot

* fix Windows custom URI passing (hopefully)

* better fix for custom uri on Linux

* upgrade Tauri for  feature

* switch url replacement order

* prevent React dev tools script being added in prod to desktop

* remove React devtools from html

* upgrade Tauri; required upgrading rspc, Axum, PCR

* pass typecheck + less cringe bigint

* clippy is love, clippy is life

* Typecheck plz

* fix bigint to number conversion

* use httpz + localhost server for Linux

* clippy be right

* Remove console.log

* [wip] proper auth

* fix Linux sidebar padding

* Secure Axum server with random

* Extracting app setup specific to linux to a different file

* remove outdated comment

* Some tweaks on cursom_uri.rs

* file_path_with_location doesn't need to be a named include

* fix typo

* factually wrong comment

* Change `unwrap` to `expect`

* bruh

---------

Co-authored-by: Ericson Soares <ericson.ds999@gmail.com>
2023-02-14 13:27:11 +08:00
Ericson "Fogo" Soares bcbcd260d4
ENG 223 Location Awareness (#468)
* Introducing location online/offline checks and location relink

* Some initial drafts for location watchers

* Location metadata relink and add library

* Many improvements at job system
Now using prisma batching at identifier job
Removing blocking I/O from extension subcrate
Implementing lazy generation of thumbnails
New current directory identifier job to be used on light rescans

* Some optimizations on identifier and object validator jobs

* merge jamie's identifier PR

* fully repaired identifier job

* properly hooked up object kind

* inspector fix

* fix video badge

* small improvements to libraries settings

* identifier and inspector improvements

* fix feature flags and hook up context menu location utilities

* BETTER CONTEXT MENU x100

* test-files

* style tweaks

* new icon designs

* manifest

* fix thumbnails on web

* media data

* New Location Watcher and some minor fixes

* disable broken media_data extractor, wip

* wip

* function name fix

* Fixing pnpm prep and some warnings

* Solving a race condition beetween indexer job and FS event handlerSome other minor warnings

* Generating thumbnails on watcher

* Remove event handler on watcher

* Some initial works on modify events and other small fixes

* File update event

* Trying to be more generic with used events and some tests to validate our assumptions

* Turning on location metadata file

* Introducing core unit tests on CI pipeline

* Submiting new unit test assumptions to validate on windows CI

* Fixing unit tests

* Fixing unit tests again

* Fixing unit tests

* Fixing unit tests for macos

* Fixing unit tests for macos again

* New structure for platform dependent event handling
Implementing event handlers for Linux and MacOS

* minor fixes + rustfmt + clippy

* Windows event handling

* Introducing a feature gate to only use location watching on desktop app for now

* Putting more stuff behind feature gates to avoid warnings

* Adding feature to cargo test on CI

* Changing some debug logs to trace logs and removing Jamie specific stuff

* Make location removal from manager less async

* fix build when "location-watcher" feature disabled

* fix types + clippy

* make location manager non-static

* remove uses of `to_string_lossy`

* more invalidate_query calls

* Expose `library_ctx` directly to avoid needless clones

* New materialized_path handling for directories

* Removing cascade delete between file_path and object
- Some other minor stuff

* remove unused `CurrentDirFileIdentifierJob`

Co-authored-by: Jamie Pine <ijamespine@me.com>
Co-authored-by: Oscar Beaumont <oscar@otbeaumont.me>
2022-12-31 00:53:24 +08:00
Brendan Allan b713c896a0
More Sync Stuff (#438)
* broken sync example

* move to react

* filter owned ops more

* update deps

* working sync

* relation operations

* fix pnpm lock

* please rustfmt

* fix api

* compare strings properly

* please clippy

* feature gate logging

* use bleeding rspc in example

* use bleeding rspc

* use generated rspc client
2022-10-27 23:12:28 -07:00
Brendan Allan 723058e060 merge main into update-rspc-and-more 2022-10-05 08:58:03 +08:00
Brendan Allan ed06e3051e
[ENG-84] Sync library (#394)
* new sql lib stuff

* add sync stuff + rename all crates

* build sd-core

* add sync/example/dist to source

* fix sync example in monorepop

* appease clippy

* update lockfile

* update commit hooks

* fix typescript

* fix typescript build

* please rustfmt
2022-10-04 05:25:12 -07:00
Brendan Allan e3f92ab4ec please rustfmt 2022-10-04 20:07:05 +08:00
Oscar Beaumont 60e228ec94 upgrade rspc 2022-10-02 01:16:11 +08:00
Ericson "Fogo" Soares 6fd620087b
Video thumbnails (#376)
* Preparing some scaffolding for video thumbnails

* Implemented thumbnail generation for videos
* Propagating errors of `Node` creation
* Using ffmpeg feature gate

* Introducing ffmpegthumbnailer-rs as a subcrate on core

* - rename to thumbnailer
- fix explorer thumbnail bug
- add more supported video types
- re-fix explorer performance

* remove nested licence

Co-authored-by: Jamie Pine <ijamespine@me.com>
2022-09-24 01:21:13 -07:00
Oscar Beaumont ef59aa40e8 Merge remote-tracking branch 'origin/main' into eng-237-explorer-performance-2 2022-09-15 12:45:05 +08:00
Oscar Beaumont dd04625a4f properly shutdown the core when using server 2022-09-14 19:02:11 +08:00
Oscar Beaumont 3caf844a9e fix the production build issue + cleanup frontend 2022-09-09 06:29:47 +08:00
Oscar Beaumont ea92af57f8
Move Rust backend to rspc (#345)
* move Rust backend to rspc

* move server to Axum + remove parts of old bridge

* move frontend over to @rspc/client

* move core to rspc + update deps

* fix Typescript errors + upgrade deps

* document invalidate_query! macro

* general cleanup + upgrade to PCR 0.6.0

* prisma error handling

* upgrade to rspc 0.0.4

* update vite-plugin-ssr

* fix typescript

* fix builds

* put landing page app name back

* hardcode ffmpeg version on Windows

* rename 'command' to 'mutation' to line up with react-query terminology

* upgrade rspc to v0.0.5 + fix types

* use shared ffmpeg binaries

* general Typescript cleanup

* fix clippy workflow failing due to tauri proc-macro

Co-authored-by: Brendan Allan <brendonovich@outlook.com>
2022-08-03 08:36:03 -07:00
Ericson "Fogo" Soares 85e5eec993
Resumable Jobs + Lefthook Integration (#344)
* Introducing Lefthook for git hooks automation
* TypeScript typechecking and linting
* Rust formatting and linting
* Spellchecking (also corrected spell errors in some files)
* Check links in md files

* Introducing resumable jobs
* Abstractions to pause and resume jobs automatically when application exits and is started
* Changing database to use Bytes for UUID fields
* Changing uuid fields on core to use uuid::Uuid instead of String
* Updating some dependencies and introducing msg_pack serialization to save job state on database

* Fixing some clippy warnings

* Fixing a regression introduced on identifier job, doing too much db accesses concurrently
2022-07-27 00:06:34 -07:00
Oscar Beaumont d7794bad4e make the Socket have named fields for clarity 2022-07-12 19:33:04 +08:00
Oscar Beaumont 7e9d036aee allow events to work from web 2022-07-11 16:38:02 +08:00
Oscar Beaumont c685ce5fe9
Library manager (#258) 2022-07-10 19:05:24 -07:00
Jamie Pine 9870d58b35 much 2022-05-10 09:03:20 -07:00
Oscar Beaumont d834fe3b2c feature gated p2p + update deps + pushed bindings 2022-04-27 17:22:26 +08:00
Jamie Pine 0287561f48 wip 2022-04-24 09:50:22 -07:00
Oscar Beaumont a8696d8faa
@sd/server in Docker and deploy to Kubernetes 2022-04-24 14:55:20 +08:00
Jamie Pine 93d80233d7 server transport via websocket
Co-authored-by: Oscar Beaumont <oscar@otbeaumont.me>
Co-authored-by: Brendan Allan <brendonovich@outlook.com>
2022-04-23 06:50:17 -07:00
Jamie Pine 40cc780ec1 added unique file identifier 2022-04-23 01:12:35 -07:00
Jamie Pine 4c4325619f server start, landing tweaks 2022-04-20 23:35:48 -07:00