Commit graph

263 commits

Author SHA1 Message Date
Brendan Allan 918c2a987d
Batch ingest sync operations (#2378)
batch ingest sync operations
2024-04-24 08:26:50 +00:00
Oscar Beaumont 643bd3a142
Block size (#2377)
Block size + some Clippy
2024-04-24 08:27:31 +00:00
Brendan Allan 663d1e9f1e
Group create and initial update operations into Create (#2329)
* ignore update after delete in sync

* Group create and initial update operations into Create

* fix ts

* fix sync generator

* fix sync debug route

* oops
2024-04-18 03:53:16 +00:00
Brendan Allan a315dd632d
ignore update after delete in sync (#2327) 2024-04-17 04:57:45 +00:00
jake 10df655fba
Remove soon-to-be deprecated clippy.toml in the crypto crate (#2322)
fix(crypto): remove soon-to-be deprecated `clippy.toml` in favour of clippy key in `Cargo.toml`
2024-04-15 12:35:07 +00:00
Oscar Beaumont 2848782e8e
OpenDAL - Ephemeral Locations (#2283)
* `sd-indexer` crate

* more wip

* wip

* fix

* IndexerRules + error handling

* wip

* Fix it

* Thumbnailer

* Link locations

* Workaround rspc's requiring `Sync` for streams.

* I hate this

* Sorting

* fix

* fix

* fix paths

* Fix double file extension

* Fix thumbnailer

* Clippy

* Fix indexer rules

* fix

* lockfile
2024-04-12 06:51:51 +00:00
Brendan Allan 40fa3380e5
[ENG-1722] Numeric sync model IDs (#2298)
* numeric sync model ids

* migration

* fix test compilation
2024-04-11 03:46:30 +00:00
Ericson "Fogo" Soares 6b760b0b2a
[ENG-1628] Write new indexer with the task system (#2161)
* Moving file-path-helper to a sub-crate on core

* Parallel walker tested and working

* Change inner core crate names to sd-core-* naming scheme

* Moving stuff around

* Save and Update tasks for the new indexer job

* Some initial drafts on the new job system

* More drafts on job system

* Changing walker task to a more explicit state machine
Also drafting more of job system

* More drafting on job resume

* Completed the draft on job system inner workings

* New job context abstraction to decouple library stuff from job system

* Properly use composition on task dispatcher

* First draft on indexer job

* Job serialization

* Handling ancestors in the distributed walker for indexer

* Saving computed directories sizes on a location to db

* Enable a way to check if jobs are running in a location

* Progress reporting on indexer job

* Reorganizing modules

* Shallow indexer

* Rust fmt

* Attempting windows CI fix

* Attempting windows CI fix again

* Attempting windows CI fix again
2024-04-08 14:32:44 +00:00
Arnab Chakraborty 888ab5639b
[ENG-1483] Right Click -> New File (#2276)
* feat: Right Click -> New File

Creates an empty file called `Untitled` and adds `(1) -> (n)` for n number of files with the same `Untitled` name.

* feat: Seperate into Text File and Empty File

* chore: `cargo fmt`

* fix: Forgot to type convert to String

* feat: Working creation

* fix: i18n changes

* More i18n

* All added tags now translated

* With proper types checking now!

* Fix types

* chore: formatting
2024-04-04 04:20:31 +00:00
Oscar Beaumont f2477d47d9
Remove IdentityOrRemoteIdentity (#2220)
* wip

* wip

* fix migrations

* fix

* Fix Prisma migrations + fire new app migration

---------

Co-authored-by: jake <77554505+brxken128@users.noreply.github.com>
2024-03-22 01:18:50 +00:00
Vítor Vasconcellos 08c42a36fe
[ENG-1624] E2E Tests (#2192)
* Basic cypress test

* Add e2e test command

* Add full e2e test for creating and deleting a library

* Add cypress to CI

* Fix cypress CI
 - Use arm macos for running cypress
 - Setup rust on cypress job
 - Increase cypress CI timeout

* Remove unused APPLE_SIGNING_IDENTITY envvar

* Update deps again + Try to fix Cypress CI setup

* Update deps again

* Only test default location if it exists

* Use latest v18 NodeJS in CI

* Increase minimum node version due to updated dependency
 - @typescript-eslint/eslint-plugin now requires node >=18.18

* Fix testing failing for optional default locations

* Enable video recording
 - Fix screenshot and video artifact uploading

* Fix location test
 - Use correct location name, instead of trying to derive it from the internal id
2024-03-19 20:22:17 +00:00
Vítor Vasconcellos 16354b0f72
Fix a couple of tests after #2218 (#2219)
- Fix some rust warnings
2024-03-19 02:23:42 +00:00
Brendan Allan 3afc3bd34f
fix cloud sync and operations (#2218)
* fix cloud sync and operations

* remove some logs

* cleanup
2024-03-18 08:51:16 +00:00
Oscar Beaumont 7dc0880f8b
P2P Metadata Cache + Sidebar (#2208)
* wip

* wip

* Render peers in sidebar

* clippy

* stringify on server

* don't stringify

---------

Co-authored-by: Brendan Allan <brendonovich@outlook.com>
2024-03-14 09:12:29 +00:00
Oscar Beaumont 27c6570c2f
Rename P2P crate (#2207)
`sd_p2p2` to `sd_p2p`
2024-03-14 04:38:31 +00:00
Oscar Beaumont 2ca88a5d6b
Refactor P2P Hooks (#2193)
* Basic AF server

* wip

* Add autonat to relay server

* Add autonat client + fixes

* Deploy script

* wip

* Debug view

* wip

* wip

* relay all events

* wip

* fix

* wip

* libp2p man spoke

* dctur

* Relay config file

* Advertise relay server

* Dynamic relay configuration

* wip

* p2p relay config

* cleanup

* push instances into p2p state

* fix

* Fix up TS

* refactor p2p hooks

* fix backend state

* Skip self

* a

* b

* Relay config in debug query

* Fix method name

* tsc is broken on my machine

* a

* Incorrect typecasts. Can we just ban them

* fix types
2024-03-13 06:43:22 +00:00
Brendan Allan 79530f1e4e
Fix sync tests (#2194)
* Fix sync tests

* fix build
2024-03-13 05:34:14 +00:00
jake 471db4a691
Clippy the entire repo (#2195)
* fix(clippy): for the entire repo

* fix: make function non-const fn
2024-03-12 11:27:23 +00:00
Oscar Beaumont 926ae4fc65
P2P holepunching + relay (#2147)
* Basic AF server

* wip

* Add autonat to relay server

* Add autonat client + fixes

* Deploy script

* wip

* Debug view

* wip

* wip

* relay all events

* wip

* fix

* wip

* libp2p man spoke

* dctur

* Relay config file

* Advertise relay server

* Dynamic relay configuration

* wip

* p2p relay config

* cleanup

* push instances into p2p state

* fix

* Fix up TS
2024-03-12 06:18:58 +00:00
Brendan Allan d1fa6af7be
Add tests for CompressedCRDTOperation (#2189) 2024-03-12 00:39:57 +00:00
Brendan Allan 62d22c9295
move CompressedCRDTOperation and add docs around sync (#2188) 2024-03-11 09:35:53 +00:00
Julian Braha fdf1c18dd6
p2p-block: check for out-of-bounds indexing (#2187)
Return err on index OOB instead of panicking
2024-03-11 06:41:14 +00:00
jake 6ed0d5edfb
[ENG-1668] Update resvg and related code (#2177)
deps: update `resvg` and related code
2024-03-07 18:22:04 +00:00
jake 355ac191b8
[ENG-440] Crypto Refactor (#2115)
* rebase: `crates/crypto` into current `main`

* refactor: remove `mnemonic` module

* feat: disable secure erase temporarily

* fix: tsc

* fix: tsc due to unused import

* fix: remove `cli` crypto info

* deps: update

* chore: remove dead comment

* refactor: remove `bincode` feature

* refactor: give `keyring` a dedicated feature so it's not reliant on `sys` as well

* fix: remove `aes-gcm` as it's no longer supported

* refactor: remove dead comment

* fix: update `keyring` imports

* refactor: change tests to `aes-256-gcm`

* feat: make `Key` a `Box<>` internally to ensure it's heap allocated (and fix tests)

* chore: clippy

* fix: hashing tests now that `const` keys aren't available

this will be cleaned up with test vectors and `include_bytes!()`

* chore: clippy

* refactor: remove dead code

* test: bring back `encrypt_with_invalid_nonce` test

* fix: secret service keyring

* fix: `zbus` build issues

* doc: update comment for clearer reasoning

* fix: cargo fmt

* fix: use bytes directly

* deps: update lockfile

* fix: secret service keyring

* fix: comment out windows keyring for now

* fix: use session keyring if no keyring backend

* fix: completely remove keyring module if no keyring is available for that OS

* fix: clippy

* fix: move iimport to correct conditional compilation

* fix: fmt
2024-03-07 08:00:37 +00:00
Brendan Allan 63b17adb10
switch sync to msgpack instead of json (#2164)
* switch sync to msgpack instead of json

* use rmpv

* more rmpv

* use msgpack in cloud

* receive operations as msgpack
2024-03-06 10:15:29 +00:00
Ericson "Fogo" Soares 68ca2382fd
[ENG-1627] Move all jobs and actors to an "Old" namespace (#2157)
* Bunch of new warns due to a stronger clippy config

* Moving old processing stuff to a new namespace
Also fixing a bunch of typos through the entire codebase.

* Rustfmt
2024-03-04 18:57:35 +00:00
Brendan Allan 54af5a822f
fix doAdd cloud api and provide operations count (#2154) 2024-03-01 10:41:47 +00:00
Oscar Beaumont 2cc828c4af
Everything over P2P (#2135)
* Fix websockets over P2P + multi-node cache

* cleanup features

* Explorer over P2P demo

* wip

* wip

* working for ephemeral stuff

* Button to get out of remote viewer

* remove

* Cleanup

* more cleanup

* Fix cross-node library queries

* Files over P2P

* Drop `Header::File`

* Fix Clippy

* Don't let noobs get hacked
2024-02-29 07:46:30 +00:00
Brendan Allan e8f329de49
[ENG-1614] Remove @default from synced models (#2134)
Remove @default from synced models
2024-02-28 04:56:29 +00:00
Ericson "Fogo" Soares dba85ebac3
[ENG-1513] Better integration between Jobs and processing Actors (#1974)
* First draft on new task system

* Removing save to disk from task system

* Bunch of concurrency issues

* Solving Future impl issue when pausing tasks

* Fix cancel and abort

* Bunch of fixes on pause, suspend, resume, cancel and abort
Also better error handling on task completion for the user

* New capabilities to return an output on a task

* Introducing a simple way to linear backoff on failed steal

* Sample actor where tasks can dispatch more tasks

* Rustfmt

* Steal test to make sure

* Stale deps cleanup

* Removing unused utils

* Initial lib docs

* Docs ok

* Memory cleanup on idle

---------

Co-authored-by: Vítor Vasconcellos <vasconcellos.dev@gmail.com>
2024-02-26 19:45:58 +00:00
Brendan Allan c533d12df0
media data sync (#2102)
* basic sync operation backfill

* media data sync

* sync entry helpers

* fix sync generator

* nicer

* re-add key_id
2024-02-21 11:42:10 +00:00
Oscar Beaumont 519b1b6c46
Fix P2P not working for libraries (#2031)
* P2P Debug route

* Remove legacy peer to peer pairing process

* Fix error typo

* Sync instances with cloud

* Upgrade deps + extended instance data

* Create instance with extended metadata

* Auto sync instances

* Actually `.await`

* bruh

* sync library info

* this isn't gonna work

* only sleep cloud receiver when no more messages (#1985)

* [ENG-1567] Fix renaming (#1986)

fix rename

* only sleep cloud receiver when no more messages

* use in memory instances during cloud receive (#1995)

* use in memory instances during cloud receive

* is_empty

---------

Co-authored-by: nikec <43032218+niikeec@users.noreply.github.com>

* fix type error

* wip

* make mdns mdns better

* rebuild state

* Add hooks + listeners + discovered state

* Split into crates

* wip fixing core + wip merging Spacetime into `sd-p2p2`

* `SmartLockGuard` + `Listener`

* Make `sd-core` compile

* Reenable all operation receivers

* Fix all broken code within `sd-core`

* minor fixes found in review

* Bring in `libp2p` + restructure `sd-p2p` for the gazillion-th time

* whoops

* Compile no matter the (runtime) cost

* fixing merge issues

* wip

* a

* b

* C

* Handle port betterer

* c

* Migrate node config

* a

* no crash on startup

* wip

* a

* jdfhskjfsg

* a

* fix discovery

* a bunch of fixes

* getting Spacedrop working

* I don't get why it no worky

* debug example

* a

* wip

* wip

* removing logging from stream impl

* wip: shit is fucked

* Redo quic integration  + Spacedrop working

* Fix shutdown - deadlocks + shutdown peers

* Add Prisma migrations

* Fix shutdown

* a

* fix

* cleanup

* The lord clippy hath spoken

* disable P2P settings for now

---------

Co-authored-by: Brendan Allan <brendonovich@outlook.com>
Co-authored-by: nikec <43032218+niikeec@users.noreply.github.com>
2024-02-21 08:13:40 +00:00
Utku d54782d7a2
Bump version to 0.2.4 (#2080)
* bump

* cargo

* bump height so it doesn't block onboarding

* disable parallelism for image labeler

---------

Co-authored-by: Vítor Vasconcellos <vasconcellos.dev@gmail.com>
2024-02-14 07:40:01 +00:00
Vítor Vasconcellos 72efcc9f62
[ENG-1595] Fix some problems with the AI system (#2063)
* Fix some problems with the AI system
 - Fix downloading model using an internal rust string representation as path for the model file
 - Fix Linux loading onnx shared lib from a hardcoded path
 - Fix App should not crash when the AI system fails to start
 - Fix sd-server failing to start due to onnxruntime incorrect linking
 - Some extra clippy auto fixes

* Use latest ort

* Fix dangling sd_ai reference
 - Use entrypoint.sh to initilize container

* Fix server Dockerfile
 - Fix cargo warning

* Workaround intro video breaking onboarding for the web version

* Fix rebase
2024-02-12 17:45:17 +00:00
Brendan Allan 177b2a23d6
sync support for labels (#2070)
* more sync support for file paths + saved searches

* sync support for labels

* update sync prisma generator to support more than tags

* workey

* don't do illegal db migration

* use name as label id in explorer
2024-02-09 13:20:51 +00:00
Brendan Allan 6f28d8ec28
More sync support for file paths + saved searches (#2067)
more sync support for file paths + saved searches
2024-02-09 08:17:04 +00:00
Oscar Beaumont a30deaa144
Cloud + P2P (#1970)
* P2P Debug route

* Remove legacy peer to peer pairing process

* Fix error typo

* Sync instances with cloud

* Upgrade deps + extended instance data

* Create instance with extended metadata

* Auto sync instances

* Actually `.await`

* bruh

* sync library info

* this isn't gonna work

* only sleep cloud receiver when no more messages (#1985)

* [ENG-1567] Fix renaming (#1986)

fix rename

* only sleep cloud receiver when no more messages

* use in memory instances during cloud receive (#1995)

* use in memory instances during cloud receive

* is_empty

---------

Co-authored-by: nikec <43032218+niikeec@users.noreply.github.com>

* fix type error

* wip

* make mdns mdns better

* response

* remove renames

---------

Co-authored-by: Brendan Allan <brendonovich@outlook.com>
Co-authored-by: nikec <43032218+niikeec@users.noreply.github.com>
2024-02-02 09:31:11 +00:00
Brendan Allan c45f6d7bfa
separate cloud sync into sd-core-cloud-sync (#2025)
* separate cloud sync into sd-core-cloud-sync

* ci
2024-01-31 08:16:54 +00:00
Vítor Vasconcellos 556ddbd4f8
Port AppImage build to use appimage-builder (#1785)
* Initial port to appimage-builder

* Almost

* Fix appimage build on arm64

* Custom patch for external binaries run under appimage
 - Disable bubblewrap sandbox when running under appimage
 - Change cwd to APPDIR when running under appimage

* AppImage Works (for the first lunch, then it crashes with Stack Smash 😭)

* Fix stack smashing, now AppImage almos fully works \o/ (gstreamer is still broken)
 - Temporarily disable the volume watcher when running under appimage (Workaround for the stack smash error)
 - Wrap gnu lic version check for appimage under conditional compile for glibc targets
 - Add error handling for the justUpdatedCheck
 - Fix VITE_LANDING_ORIGIN being undefined

* On non glibc systems default to runtime/compat

* Use glibc version 2.8 instead of 0 for non-gnu systems

* Fix video playback not working due to broken GstRegistry

* Build and publish new AppImage release artifact
 - Fix model location when building deb
 - Improve model path resolution logic
 - Remove patchelf dependency from setup script
 - Fix incorrectly ignore gstreamer dependency in AppImage recipe

* Fix clippy complaining about `get_path_relative_to_exe`
 - Read GLIBC_FAKE_VERSION or use 2.8 for musl in appimage (while the code is there, this is not really supported for now)

* Remove appimage tauri target from release CI

* Remove setup-buildx-action, not relly needed

* typo fix

* Fix git describe command running on cwd instead of the repo root dir

* Attempt fix weird git permissions errors in CI+docker

* Pass CI env to docker appimage

* Only use git after installing it

* Pass target to appimage build script

* Fix permission after creating appimage

* -_-

* Swap envvar with github ci var

* Format

* Add instruction on how to manually build an AppImage

* Fix typos

* docs: add note about running podman with `--privileged` if there's a permission denied error

* docs: fix typo and link directly to appimage-building `README.md`

* refactor: streamline code and make it a bit cleaner

---------

Co-authored-by: jake <77554505+brxken128@users.noreply.github.com>
2024-01-22 10:52:26 +00:00
Julian Braha 9018b9277f
Implement Hash, PartialEq, and Eq for CacheNode (#1964) 2024-01-21 01:28:06 +00:00
Julian Braha 04d6b2646a
Add missing zeroize to Identity (#1962)
Add derive(ZeroizeOnDrop) to Identity
2024-01-20 15:15:06 +00:00
Jamie Pine fdf31fc3a8
[ENG-1511] Library Screens (#1903)
* init

* changes

* Now updating statistics once a minute

* More robust statistics updater

* Concurrency is hard

* improvements to stats

* refactor

* adjust setting back/forward padding so it matches top bar

* refactor sidebar

* rename

* setting up screens

* some changes

* Co-authored-by: Brendan Allan <Brendonovich@users.noreply.github.com>

* yes

* yes2

* refactored explorerItem.ts

* important explorer code shouldn't be thrown away in a util moment

* support for multiple thumbnails in ExplorerItem

* clippy

* move debug

* yes

* label filters

* ts

* comment out unconnected stuff

* added .mid for midi files

---------

Co-authored-by: Ericson Fogo Soares <ericson.ds999@gmail.com>
Co-authored-by: Brendan Allan <brendonovich@outlook.com>
2024-01-16 12:15:03 +00:00
Brendan Allan 4962b1160b
[ENG-1488] Cloud sync message compression (#1922)
* CRDTOperationWithoutInstance

* almost there

* fully compress messages

* implement more of sd-cloud-api

* sd-cloud-api-ify

* landing -_-

* openssl

* clippy

* idk

* bruh

* wut

* ahhh right
2024-01-09 10:21:23 +00:00
Oscar Beaumont bef1ebcade
[ENG-1508] Launch Spacedrop (#1893)
* Shit UI

* refactor a bit

* wip

* yeet

* farming the wry but it's stale

* Real-time hover event

* Hook with `Platform` + fix broken window-state plugin

* `DragAndDropDebug`

* Clippy

* revert Tauri v2 stuff

* minor

* probs not gonna work

* undo last commit

* a

* b

* c

* d

* e

* f

* g

* long shot

* 1

* no 7

* da hell

* large bruh moment

* lol

* zzzz

* SSH into CI

* yeet

* Tauri mouse position without new Wry

* go for gold

* Correctly lock `ort` version

* minor fixes

* debounce hover events

* WTF Tauri

* Replace DND hooks with goated versions

* wip frontend stuff

* fix ts

* disable library p2p stuff

* remove Spacedrop dialog + hook up backend

* `useOnDndLeave` working

* Close popover when drag outside

* Allow `openFilePickerDialog` for Spacedrop

* couple of fixes

* empty state

* smh

---------

Co-authored-by: Brendan Allan <brendonovich@outlook.com>
2024-01-08 06:42:17 +00:00
Oscar Beaumont 949707c7f9
Vite upgrades (#1911)
* Upgrade Vite + use SWC React plugin

* Upgrade to type module

* lazy load Sentry

* Lazy load prism

* fix

* Lazy load some of the icons

* fix types

* Fix eslint config

* run lint --fix

* Upgrade Turbo

* Turborepo not happy

* Upgrade Typescript

* Stop complaining Turborepo
2024-01-02 06:26:28 +00:00
Oscar Beaumont 26f610014a
Improve ephemeral walker (#1877)
* `non_indexed::walk` as `impl Stream`

* wip

* this is gooooood

* savepoint

* remove

* Batched stream

* `unsafe_streamed_query`

* nightmare nightmare nightmare nightmare nightmare nightmare

* JS Mutex

* cleanup

* proper error handling

* myCode satisfies Typescript

* Move to npm fork of rspc

* fixes

* rspc more crashy crashy

* Typescript is very disappointed
2024-01-02 06:09:44 +00:00
Julian Braha d18ab1f56e
Add missing timeouts (#1907) 2024-01-01 23:44:48 +00:00
Brendan Allan 5371e9f961
[ENG-1512] Replace sync operation types with single type (#1900)
* replace multiple op types with crdt_operation

* cleanup
2023-12-21 05:58:03 +00:00
Ericson "Fogo" Soares 7c90bcb95b
[ENG-1479] AI Prototype (#1845)
* First draft on image labeling

* Fixing execution providers for other OSs

* Better error handling and shutdown

* Working with shallow media processor

* bruh

* Fix warnings

* Now hooked to media processor job

* Link desktop app with libonnxruntime to avoid TLS error during startup

* Be able to change models on runtime
Revert to use labels table instead of tags

* A bug on a model-less inference

* Show AI labels on Inspector
 - Change yolo inference to use half precision
 - Add labels api to core

* Remove LD_PRELOAD

* Fix race condition on model executor shutdown

* Don't load all images in memory moron

* Embeed yolo model in prod build
 - Change yolo model path to new one relative to executable

* Disable volume watcher on linux, it was crashing the app
 - Invalidate labels when they are updated

* Rust fmt

* Minor changes

* Gate onnxruntime linking to the ai-models feature

* Add build script to sd-server to handle onnxruntime linking workaround

* Move AI stuff to its own crate and normalize deps

* Rust fmt

* Don't regenerate labels unless asked to

* Now blazingly fast

* Bad merge

* Fix

* Fix

* Add backend logic to download extra yolo models

* Add models api route
 - Add api call to get available model version
 - Add api call to change the model version

* Improve new model download logic
 - Add frontend to change image labeler model

* Fix new model downloader

* Fix model select width

* invalidate labels count after media_processor generates a new output

* Rename AI crate and first draft on download notifications

* fix types

---------

Co-authored-by: Vítor Vasconcellos <vasconcellos.dev@gmail.com>
Co-authored-by: Brendan Allan <brendonovich@outlook.com>
2023-12-19 09:28:57 +00:00
Brendan Allan 3cabc9c3a9
Basic actor manager (#1888)
* basic declared actor manager

* put actors in separate file

* clippy

* hopefully clean up some clippy warnings

---------

Co-authored-by: jake <77554505+brxken128@users.noreply.github.com>
2023-12-13 17:49:22 +00:00