Commit graph

288 commits

Author SHA1 Message Date
jake 0dae7f7a63
[ENG-828, ENG-633] Fix corrupt videos force crashing the app (#1028)
* fix corrupt videos crashing the app

* fix typo

* self-proclaimed genius
2023-06-27 05:04:40 +00:00
Oscar Beaumont 0f7a669e5a
[ENG-732] Pairing by library (#943)
* library keypair

* allow opening stream to non-connected node + more

* library identity

* fix types

* fix maybe undefined

* don't forgor migrations file

* library manager inside p2p manager

* rename

* `NodeInformation` struct

* node info exchange

* fill in info

* streamify tunnel

* use tunnel for p2p events

* libp2p is annoying + stop leaking private key's

* Clippy cleanup
2023-06-19 05:13:30 +00:00
Brendan Allan d683d22c82
Remove owned sync (#967)
* remove owned sync + cleanup

* no more atomic records

* byebye owned

* remove owned ops from schema
2023-06-18 23:45:06 +00:00
Oscar Beaumont fd1afd8500
[ENG-568] Spacedrop Progress (#956)
* `Transfer` struct

* progress moment

* fixes
2023-06-18 08:40:51 +00:00
Oscar Beaumont 3d224e02ad
[ENG-675] Spacedrop is 0kb (#942)
* fix sd-web

* pain

* unit tests are pog

* Ignore `sdserver_data2`

* Update mod.rs
2023-06-15 00:32:54 +00:00
Ericson "Fogo" Soares e693c7a542
[ENG-381] Update context menufs actions to support multiple source (#927)
* Generalizing filesystem jobs to accept multiple files at once

* Some small changes and fixing front

* Enable rename route to replicate Finder's behavior

* Assign tag to multiple objects at once

* Remove from recents accepting multiple files

* Fixing some warnings

* Adding multiple files feature to Open and Open With

* Conditional stuff for macos

* Generating commands.ts and other minor warnings

* Rust fmt

* TS typecheck

* Rust format and TS typecheck

* Requested changes

* Requested changes

---------

Co-authored-by: Utku <74243531+utkubakir@users.noreply.github.com>
2023-06-14 23:00:28 +00:00
Oscar Beaumont 049966d848
[ENG-674] fix invalidation (#891)
* reenable the invalidate requests system

* a tester for the invalidate system

* watch locations in Library::load

---------

Co-authored-by: Brendan Allan <brendonovich@outlook.com>
2023-05-31 08:55:46 +00:00
Oscar Beaumont e4696c7303
[ENG-667] Fix Clippy CI (#887)
* plz just work

* no shot

* just disable Tauri notifications

* Tauri is weird AF
2023-05-30 17:11:37 +00:00
Oscar Beaumont 5f71e0c1f6
[ENG-546] Improve error handling (#802)
* log to disk

* remove some unwraps

* panicless

* some p2p error handling

* clippy moment

* Fix `<ErrorBoundary />`

* open logs button

* 39 to 0

* fix types

* update deps and comment out broken tests

* clippy

* more clippy

* upgrade rimraf - https://github.com/isaacs/rimraf/issues/259

* regen broken lockfile

* update `notify` and update `commands.ts`

* more clippy (pls work)

* allow deprecated for p2p (hopefully temporary)

* a

* upgrade deps for p2p

* do betterer

* do it correctly

* remove unused import

* improve core startup error + bc keypair

---------

Co-authored-by: Utku Bakir <74243531+utkubakir@users.noreply.github.com>
Co-authored-by: brxken128 <77554505+brxken128@users.noreply.github.com>
2023-05-30 09:24:05 +00:00
Brendan Allan 6710f05251
move prisma codegen into sd-prisma (#876) 2023-05-29 08:37:58 +00:00
Utku dbec988147
[ENG-362, ENG-476, ENG-503] Next.js (#817)
* update landing packages & tailwind

* move images

* refactor to nextjs

* doc stuff

* rename markdown to mdx

* working docs

* Add math

* fix pnpm-lock

* Code highlighting

* fix types

* fix more stuff

* Notice component and removed slot

* delete editorconfig eslint rule

* pnpm-lock

* some fixes

* fixed types (pls)

* bump ui's typescript too

* commit next-env.d.ts

* fix doc links

* Use next/head instead of react-helmet & rehype-external-links

* fixes

* fix unsub from list

* add trim() to docs.excerpt

* trim doc title too

* fix titles

* replace <a> to Link

* hide .contentlayer & .next from vscode search
2023-05-21 13:11:03 +00:00
jake fb95c6fec2
HEIF Fixes and Improvements (#838)
* limit HEIF to macos-only because of ubuntu's *archaic* `libheif` version

* drop png entirely, go from raw `[r, g, b...]` -> `DynamicImage`
2023-05-20 22:08:04 +00:00
Jamie Pine 7e4c58a91c
[ENG-603] Support HEIC images (#834)
* working HEIC thumbnails

* better error handling

* better error handling and some cleanup

* fix type of maximum size, and clippy

* better extension support, WIP quick preview, better image resolution

* remove unnecessary x86_64 macos rustflags

* add correct rustflags to setup script

* add fedora libheif deps

* debian libheif deps

* arch libheif too

* add sd-heif as a dep and feature gate it

* enable aforementioned feature in tauri only

* add URI support for heif/heic (quick preview still won't work)

* correct feature gating on everything

* dedicated sd-heif crate

---------

Co-authored-by: brxken128 <77554505+brxken128@users.noreply.github.com>
2023-05-20 06:40:37 +00:00
jake 1aa8826398
SPDX license and workspace inheritence for re-used metadata (#833)
use workspace inheritence for license, edition and repo URL
2023-05-20 01:52:55 +00:00
Oscar Beaumont 01e8716b94
[ENG-559] Spacedrop move to spaceblock (#824)
* Move Spacedrop to Spaceblock

* bruh

* unit test moment

* plz work

* usize != u16 man

* tsc --quite

* clippy
2023-05-18 19:02:44 +00:00
Oscar Beaumont dd3a0ff689
P2P Stream ids (#630)
inbound stream id
2023-05-18 06:54:20 +00:00
Jamie Pine e8895c1a05
[ENG-581, ENG-597, ENG-600] Overview categories (#779)
* added base UI for categories on overview

* update core

* cleanup ui

* Inspector default view if nothing is selected, explorer takes child components, hidden menu tweak if no items are sm:flex

* wip

* somewhat functional

* scroll

* category fixes

* clean category bar

* added config store + made toolbar available on all explorer screens

* clean up overview.tsx

* added counts

* fix inspector bug

* add support for favorites
+ add book extension support

* refactor into smaller components

* Some small rust nitpicks

* fix camel case location_type

* Rust fmt

* fix typescript CI

---------

Co-authored-by: ameer2468 <33054370+ameer2468@users.noreply.github.com>
Co-authored-by: nikec <nikec.job@gmail.com>
Co-authored-by: Ericson Soares <ericson.ds999@gmail.com>
2023-05-17 20:34:05 +00:00
Ericson "Fogo" Soares dda7516980
[ENG-536] Improve indexer for huge locations (#781)
* WIP materialized_path abstraction revamp

* Optimizing indexer rules loading

* Using a better serialize impl for indexer rules

* New interruptable and faster Walker

* WIP new indexer

* WIP first success compiling after breaking the world

* Fixing some warnings

* Handling some lifetime issues in the walker

* New job completed with errors feature

* Introducing completed with errors to indexer
Removing IOError variant from JobError and using FileIOError instead

* Rust fmt

* Adding missing job status

* Better ergonomics to IsolatedFilePathData
Conversions from db's file_path kinds
Keeping original's relative path data to better conversion to OS's path
First unit tests

* Testing and fixing parent method

* Some error handling

* Rust fmt

* Some small fixes

* Fixing indexer rules decoding

* Bunch of small fixes

* Rust fmt

* Fixing indexer rules

* Updating frontend to new materialized_path format

* Trying to fix windows CI

---------

Co-authored-by: Brendan Allan <brendonovich@outlook.com>
2023-05-16 14:15:51 +00:00
Vítor Vasconcellos cf482f5621
[ENG-270, ENG-213] Stabilize release pipeline (#768)
* Attempt at building our own ffmpeg dylibs for macOS
 - Create dockerfile to setup osxcross and required host dependencies for cross-compiling ffmpeg for macOS
 - Create script to setup required macOS dependencies and build ffmpeg for both x86_64 and aarch64

* Improve dockerfile to better use docker's cache
 - Attempt at fixing arm64 build (still not there yet)

* ARM64 sucessfully compiles \o/
 - Use clang for everything (e.g. linker, assembler, ...)
 - Remove -lc++ because it is now handled internally thanks to a patch
 - Apply 3 open PRs to osxcross as external patches to solve some of our problems
 - PR 180 reduces the macports dependencies size by prefering non universal libs for x88
 - PR 314 adds the -lc++ flag for macOS SDKs >= 11.1
 - PR 379 fix macports pulling incorrects deps for SDK11 and adds support for SDK13 (future proofing, as it does nothing for us rigth now)

* Fix relink and copy logic in ffmpeg build script
 - Ensure ffmpeg and dependencies dylibs are correctly relinked and copied over to the output directory
 - Copy includes from ffmpeg and dependencies to allow for dev building against these dylibs
 - Set up symlinks for all dylibs to the common `lib` directory
 - Remove superfluous linker and compiler options
 - Add `-headerpad_max_install_names` linker option to accommodate the required relink changes
 - Remove `--disable-fft` for the arm64 build, as it doesn't seem to be required nowadays
 - Allow `swresample` and `avdevices` to be compiled as they seem to be expected to exist by `ffmpeg-sys-next`
 - Disable ffmpeg program compilation
 - Fix dylibs id to match their new expected location
 - Fix ffmpeg dylibs inter-linkage not being updated
 - Add custom patch for osxcross macports to load already downloaded files from cache
 - Remove the requirement for specifying the darwin version in the Dockerfile
 - Enable docker cache for macports
 - Move arm min macOS version ARG to the top

* Attempt at building ffmpeg with github actions
 - Split osxcross setup into it's own Dockerfile to allow exporting it as a image to Dockerhub
 - Create a github action to build ffmpeg
 - Add osxcross patch for PR 372 for future proofing (Add support for macOS SDK 13.1)

* Remove incompatible step name

* Change FFMPEG_VERSION to string to avoid yaml weird type casting

* Start to use our ffmpeg build in the macOS
 - Adjust the macOS setup script to download our ffmpeg build, also replace installing protobuf compiler from brew with grabbing it from it's own repository
 - Some misc improvement to the *nix setup script
 - Fix ffmpeg build script not copying static libs (for when we may want to statically link ffmpeg)
 - Fix ffmpeg build script not taking into acount possible libraries that are multiple paths deep in /lib
 - Update CI to actions/checkout@v3, v2 is deprecated
 - Update ffmpeg build CI to not xz the artifacts, as github already zips it anyway

* Fix macOS setup not finding the ffmpeg artifact due to a jq script mistake
 - Comment out embeeding static libs in ffmpeg artifact for now, as they take too much space
 - Remove incorrect dockerfile settings from editorconfig

* Fix protobuf download in macOS
 - Fix Framework directory creation not following script's dir but cwd
 - Improve download logic to avoid being rate limited on failures
 - Return to using xz for ffmpeg build, as it saves quite a bit of space

* Replace `OSTYPE` with `$(uname)` for a more standardized way to check which os is running
 - Fix mobile check using `==` instead of `=` in test command

* Fix syntax error in bash 3.x (which is the default in macOS)
 - Fix jq not downloading all deps and being placed in the incorrect directory

* Fix incorrect install_name_tool arguments
 - Remove leading './' when referencing ffmpeg libs during build

* Fix protoc not being executable
 - Fix some error messages

* jq throw input when it is invalid
 - Enable debug log in setup script when running in CI

* Add a dev.js build script for desktop app
 - Remove dll copy from setup-system.ps1, now done by dev.js
 - Minor rework to build.js to improve signals and stdio handling

* Configure cargo to use FFMPEG_DIR and PROTOC envvars on macOS

* Pass GITHUB_TOKEN to macOS setup script in CI

* Replace simple curl with gh_curl in github request

* Fix some ctrl+c/ctrl+v typos

* Add missing ffmpeg deps
 - Fix missing symlinks for ffmpeg libs in the output_dir/libs
 - Change arm64 minimum macOS target to 11.2 due to libbrotli only being available precompiled to this version
 - Add more options to ffmpeg configure in an attempt to reduce its size + improve performance

* Rework ffmpeg build script to create a FFMpeg.framework instead of loose dylibs
 - Disable unused postproc and enabled missing required protocols in ffmpeg configuration
 - Adjust Dockerfile and setup-script to handle the new FFMpeg.framework

* Fix setup script incorrect linking logic for FFMpeg.framework

* Forgot to remote the leading path in the lib/header linking logic for FFMpeg.framework

* Enable size optimization for ffmpeg
 - Remove Frameworks directory before setting it up in setup-system.sh

* Revert lib id to use `@executable_path/../Frameworks/` instead of `@rpath`
 - `@rpath` breaks dev builds
 - Add logic to create less versioned versions of dylibs in setup-system script, due to it being required for compiling ffmpeg-sys-next (not required during runtime, just for compilation)
 - Add scripts/deps to gitignore

* Fix ffmpeg build script not copying unversioned symlinks to FFMpeg.framework
 - Remove unecessary unversioned symlink logic from setup-system
 - Add new build scripts for dev and prod that setup the environment and tauri to correctly point to FFMpeg.framework
 - Rework windows setup-system script to follow the same logic as in macOS setup-system

* Attempt at fixing windows setup script
 - Add powershell editorconfig

* Attempt to fix Windows CI

* Fix some mistakes in the build scripts
 - Replace toml lib with something more used and maybe better?
 - Attempt to fix Windows CI

* Fix some more problems with the build script

* Another attempt at fixing windows CI

* Some more fixes for the windows setup script

* Add build.js step to release CI
 - Implement patch only mode for build.js when running under CI
 - Implement appeding extra env in cargo config in env.js
 - Only run windows dlls copy in dev in env.js
 - Another attempt at fixing windows CI

* Maybe this will make the windows CI happy?

* Windows CI why dont you work, whyyy?

* Try to apease the Windows CI Gods
 - Disable updater build in macOS, due to tauri-apps/tauri#3933

* Fix build.js extra env vars
 - Uncomment parts of setup-system.ps1 to check if it keeps working

* Uncomment another section and see if windows CI will complain

* uncomment some more of setup-system.ps1

* Use Start-Process for robocopy instead of executing it directly
 - Only raise an error if robocopy exists with a status code >=8
 - Revert build.js macOS specific bundles change
 - Disable updater for now

* Remove updater from cargo.toml

* Add -PassThru to Start-Process to ensure we get and Process obj back

* Comment out more of updater

* Improve documentation for both of the docker containers
 - Fix a harcoded install_name_tool call in the ffmpeg-build-macos.sh script

* Configure tauri to copy ffmpeg dlls to windows prod version
 - Fix windows dlls copy on dev
 - Stop replacing tauri.conf.json, except on CI, for prod builds and use a tauri.conf.patch.json for any required changes
 - Don't unset macOSPrivateApi on prod build, we need it for the transparent background
 - Fix dev.js not exiting when the spawned command was over

* Fix windows not find protoc
 - Fix missing import in env.js

* Fix Windows Dlls copy logic for prod build
 - Fix dev script failing due to missing env
 - Implement error handler to spawned process

* Format + Fix pnpm-lock

* Fix video thumb generation failing on Windows due to bad path to CString conversion logic

* Fix mobile build
 - Greatly improve windows setup-script error handling
 - Install LLVM with winget instead of downloading exe from github

* Improve CI a bit
 - Fix mobile CI not using github token to avoid being rate limited

* Fix pnpm not accesible to actions/setup-node

* Skip pnpm check on CI

* Fix pnpm skip not working
 - Fix pnpm cache check missing a step

* Only lock action to major versions

* CI really doesn't like running robocopy directly

* Attempt to build our own patched tauri bundler

* Trigger new workflow

* Fix attempt for patched tauri action tests
 - Disable cache for patched tauri action
 - Enable package publishing for patched tauri action

* Attempt fix patched tauri publish

* Remove commented code from patched tauri workflow
 - Replace deprecated `actions-rs/toolchain` with `dtolnay/rust-toolchain`

* Fix url patching logic for publishing step of patched tauri package

* Fix prepublishOnly command for patched tauri package

* Adjust patched tauri package package.json

* Also patch the specific os/arch sub packages

* Instruct napi to stop doing github releases

* Patch package.json befor build

* Re-enable updater
 - Patch tauri cli to workaround a tauri bug that prevented use from bundling frameworks in the updater
 - Update tauri/api to 1.3 and tauri/cli 1.3.1
 - Add target triple in the release artifacts name
 - Use ubuntu-20.04 instead of ubuntu-latest
 - Configure release workflow to build to all supported platforms
 - Replace `tauri-apps/tauri-action` with simply calling `pnpm desktop build`
 - Simplify tauri-patched-cli-js to only build the required macOS binaries
 - Unify build.js and dev.js in a tauri.js build script, which now acts as an all around wrapper for the tauri/cli

* Restore incorrectly removed step from `tauri-patched-cli-js`

* Disable auto-run of tauri-patched-cli-js
 - Update setup-system.sh with the latest tauri-patched-cli-js artifacts

* Forgot to mkdir bin

* Fix pnpm-lock.yaml not being up to date
 - Remove pnpm deps caching

* Fix variable name conflict

* Enable the updater build conditionally
 - Enable release workflow to test it

* Fix artifacts not being published
 - Change macOS on arm64 minimumSystemVersion to 11.2
 - Commented out rust cache to check if this is the culprit to the low memory problem on windows runner
 - Add some debug log to tauri.js

* Fix release workflow artifact uploading

* Specify which bundles to build on each platform
 - Attempt fix apple arm build
 - Don't include internal deb archives in linux updater artifact

* Attempt to fix apple arm build
 - Fix dmg background not applying

* Fix incorrect semver usage

* Patch swift-rs to see if that fix apple arm64 release build

* Update swift-rs submodule
 - Remove unused deps lodash.merge

* Configure so only release workflow runs without cache
 - Improve tauri.js logs

* Ignore `crates/swift-rs`

* Revert "Ignore `crates/swift-rs`"
- Remove `crates/swift-rs` before running `cargo fmt`

This reverts commit 851bd84373.

* Github CI input/output are always string
 - Attempt to fix Windows Clippy

* Fix CI syntax error

* Fix mobile CI
 - Disable pnpm deps caching
 - Disable rust targets caching on Windows
 - Configure cache factory to run for all targets

* Remove `crates/swift-rs`

* Attempt fix Mobile CI
 - Enable cache factory run in pull_request to test it
 - Specific more path for the CI run to ignore
 - Specify path that will trigger the cache factory run
 - Some cleanup in the setup-pnpm action

* Remove restore-cache options
 - Add prefix key to cache-factory
 - Fix LLVM cache-hit check
 - Add libheif to FFMpeg.framework

* Remove submodule clone from actions

* Add fake deps to ffmpeg build dockerfile
 - Fake install deps that aren't really necessary to build ffmpeg
 - Fail ffmpeg build script if it can't find a macports deps
 - Copy libheif headers to Framework

* Remove automatic trigger for relase and pull request trigger for cache-factory actions

* Remove pnpm setup from Cache factory
 - Cache factory is exclusivly for rust deps caching, no need to setup Node and pnpm with it

* Re-enabled fail-fast in Github CI
2023-05-15 18:02:56 +00:00
jake 207c82bfeb
[ENG-590] Uppercase file extensions are "unknown" (#801)
match against the lowercase extension
2023-05-07 17:58:19 +00:00
Oscar Beaumont 2571e3b275
Upgrade rspc TS (#775)
* finally

* should be working?

* fix types

* fix types

* Wouldn't it be nice if Metro would just work

* idk

* try harder Metro

* potentially fix bundling issues

* idk, maybe fix it?

* fix metro

* update podfile.lock

* bruh

* bruhz

* tailwind is drunk again

---------

Co-authored-by: Utku Bakir <74243531+utkubakir@users.noreply.github.com>
2023-05-04 06:21:42 +00:00
Brendan Allan 7cee2b2651
update prisma (#773)
* update prisma

* idk
2023-04-30 23:03:23 -07:00
Brendan Allan e1c626c0fa
Remove all field name string literals (#772)
remove all field name string literals
2023-04-28 21:59:11 -07:00
Brendan Allan 82845a8388
Upgrade Prisma (#770)
upgrade prisma
2023-04-28 17:56:08 +00:00
Oscar Beaumont 3c440783b3
Spacedrop (#761)
* A hint of file drop

* backport from #671

* accept/reject Spacedrop working

* final cleanup

* Rename spacedrop.tsx to Spacedrop.tsx

* Update index.tsx
2023-04-27 04:41:33 +00:00
Oscar Beaumont 85d0ddf6d3
rspc 0.1.3 prep (#756)
* Upgrade rspc + Specta + PCR

* fix jsonrpc executor

* fix invalidate excessive logging

* fix `Protected` type

* fix mobile

* clippy

* lol
2023-04-25 15:14:52 +08:00
jake 37b8d64642
[ENG-490] Add support for late key manager setup (#740)
* support for late key manager setup and remove password from onboarding

* mobile

* delete master-password.tsx

* remove secure temp keystore

* remove more km_config artifacts

* fix builds

* Fix most of the UI, there still some minor bugs

* Workaround portalled closing Popovers

* Restore search in core/src/api/mod.rs

* Fix Password eye icon closing Popover on click

* Fix keymanager setup submit button spinner not centered
 - Fix keymanager setup missing logic for disabling form
 - Adjust keymanager tab roundness to match app style

* Restore .gitignore

* Restore .prettierignore

* fixerino tests

---------

Co-authored-by: Utku Bakir <74243531+utkubakir@users.noreply.github.com>
Co-authored-by: Vítor Vasconcellos <vasconcellos.dev@gmail.com>
2023-04-22 09:10:21 +00:00
jake 4315cee147
[ENG-516] Prevent copies/cuts to the same source/dest (#748)
* add basic UI protection to stop cut/copies to the same path

* add rust src/dest protections

* fix pasting logic

* fix path canonicalization

* skip paths instead of overwriting them

* Using non-blocking ops and fixing some warnings

---------

Co-authored-by: Ericson Soares <ericson.ds999@gmail.com>
2023-04-22 02:11:59 +00:00
Brendan Allan 469a3831dc
Explorer search (#738)
* basic file path search

* cleanup

* render explorer in search page

* update materialized path docs
2023-04-21 14:49:46 +00:00
jake 288f4947e1
remove int-enum (#721) 2023-04-19 01:33:32 +00:00
Vítor Vasconcellos 03eb27e91d
[ENG-469] Make Prettier and ESLint work together (#706)
* Make Prettier and ESLint work together
- Resolve conflicts between Prettier and ESLint regarding indentation and Tailwind rules order
- Add `.editorconfig` to standardize basic formatting options across tools and editors
- Add `.gitattributes` to hide `pnpm-lock.yaml` in `git diff` output
- Include EditorConfig in the recommended extensions for VSCode
- Replace some instances of `pnpm exec <command>` with `pnpm <command>`
- Remove superfluous Tauri config for Linux

* Revert Prettier changes (it was working correctly before)
 - Update ESLint to read Tailwind config from absolute path
 - Remove redundant Prettier dependency from subprojects
 - Specify the source folder for the lint script in subprojects

* use mobile's tailwind config with eslint

* pnpm format + pnpm lint:fix

---------

Co-authored-by: Utku Bakir <74243531+utkubakir@users.noreply.github.com>
2023-04-14 21:21:21 +00:00
jake 39e92cc7e7
update to ffmpeg 6.0.0 (#697) 2023-04-13 14:10:23 +00:00
Oscar Beaumont 3d6afe5447
[ENG-414] P2P Shutdown (#688)
* `Event::Shutdown` in P2P system

* Mdns shutdown broadcast + await shutdown

* Fix mdns service name & prevent poll after shutdown
2023-04-11 15:05:50 +00:00
Oscar Beaumont 4c0e341e55
Add MetadataManager into sd-p2p (#686)
* Add `MetadataManager` into sd-p2p

* Add `update_metadata` helper to P2P manager

* Praise Clippy our lord and saviour

* change mdns advertisement log level

* Casually broke Linux
2023-04-11 10:20:14 +08:00
Ericson "Fogo" Soares b346e7ac52
Eng 372 location awareness improvements (#612)
* Complying with a pedantic Clippy

* Solving duplicated directories events
When creating a directory through MacOS's Finder, for some reason
FSEvents receives 2 Create Folder events that we have to handle

* Handling moving to trash bin and restoring on Mac
Still missing the feature to restore a directory and its children

* Now handling creation of empty files on MacOS

* Enabling restore of directories and its children

* Now working: moving a directory to another
inside the same location

* Now Indexer also remove file_paths not on fs

* Enabling multiple file moves on location watcher

* Fix Windows conditional compilation issues

* Fixing cas_id generation and bumping some deps

* Many Windows specific improvs and some refactors

* Rust fmt

* Using conditional compilation on extract inode function

* Linux fixes and some MaterializedPath improvements

* Rust fmt again

* Introducing tick behavior on location watchers

* Making LastFilePathIdManager more atomic

* Some vscode launch configs for lldb debugger

* Simplifying some lifetimes

* Making all watchers more consistent
2023-03-31 04:59:33 +00:00
Oscar Beaumont c2666f7f63
Sync over p2p (#606)
* sync no networkey

* sync technically wired up

* fix p2p endianness for sync payload

* some sync debug stuff

* feature gate message production

* bump lockfile

* execute queries when sync messages disabled

---------

Co-authored-by: Brendan Allan <brendonovich@outlook.com>
2023-03-21 06:03:08 -07:00
Brendan Allan 145f87cdab
Data Router (#609)
* data router time

* update pnpm.lock
2023-03-15 16:11:47 +00:00
jake e12475a4e1
[ENG-430] *Nix setup script improvements (#602)
* add `protobuf` deps to arch and fedora

* update webkit2gtk package name for fedora >= 37

* revert last change as fedora 37/38 can still use the old name

This reverts commit cc9b2d7053.

* shfmt

* re-introduce webkit changes (and fix my bad bash)

* remove duplicate webkit dep

* update mac setup logic

* remove ffmpeg versioning (we don't use it on any other system)

* check that RPMFusion is enabled

* more verbose protobuf variable names

* add comments/todos and remove unnecessary 64 bit declaration

* fix incorrect command & use latest ffmpeg on macos

* update deps & remove comments
2023-03-15 16:11:15 +00:00
jake 171f19bc71
[ENG-429] Crypto organization/refactor (#607)
* update comment

* key manager feature gating

* update deps & more feature gating

* fix keyring feature gating

* add an `exhaustive_read` function for crypto/stream

* restructure STREAM module

* move tests+shared utils

* clean up stream decryption

* further cleanup

* impl to `GenericArray` for `Nonce`

* update examples

* update refs & `use`s

* fix `Nonce` -> `GenericArray` conversions

* better `Protected` conversions + remove `Password` type

* a work of art

* finishing touches

* some API changes

* rename `StreamX` to `X`

* fix everything else

* separate `primitives` from `types`

* update imports & fix build
2023-03-15 16:09:36 +00:00
Brendan Allan 53229809ca
update swift-rs (#605) 2023-03-13 07:53:28 -07:00
Oscar Beaumont 1978e2ff48
Eng 432 refactor mdns service/state (#604)
clean up mdns state and refactor advertisement system
2023-03-13 18:05:46 +08:00
Oscar Beaumont 0c25239c53
Typesafe stream handling (#603)
* typesafe stream handling
Now it's impossible for your to respond to a broadcast stream. Previously this would just fail due to the TCP connection having been closed by the sender.

* remove connected_peers state from `SpaceTime`

* use `OutEvent` instead of `tokio::spawn` for `SpaceTime` events

* Polling for events > tokio::mpsc

* extend max length of sync messages to ~4GB + broadcast done message

* Fix "actOS" operating system decode error
2023-03-13 16:58:35 +08:00
Ericson "Fogo" Soares 4a6b057872
[ENG-371] Subpath shallow re indexing (#599)
* Introducing sub path indexing for IndexerJob

* Introducing shallow indexer job

* Some feature flags to avoid warnings at pnpm prep

* Many fixes at shallow indexer job

* Changing how we implement the bookkeeping for file_path ids
Now we're account for independent locations, and also integrating
better with the LibraryContext, instead of using a static global

* Making indexer job reentrant

* Introducing shallow file identifier job
And a bunch of minor refactors

* Rust fmt

* Removing light scan from getExplorerData query
Light scan is a mutation, so we can call it on useEffect function from
the Explorer component in the frontend, when a location_id or the explorer
path changes

* Handling job early finish on init for identifier

* Only invalidate query if we have orphan paths

* Introducing ShalowThumbnailerJob

* Clippy warnings about `into_iter()`

* Naming scheme for Prisma's selects and includes

* Invalidating getExplorerData at thumbnailer

* Small mistakes lol

* Some nitpicks with pnpm prep

* Rust fmt

* Changing indexer's walk log to `trace!`

* Not deleting all file_paths on location fullRescan

* TS typecheck

* Removing `file_path` selection just with id
2023-03-10 16:25:58 +00:00
Oscar Beaumont 7c5f760fbd
Basic Spacedrop working with rspc! (#601) 2023-03-09 06:23:17 -08:00
Jamie Pine f98d7ad6e0
[ENG-408] Dynamic Object icons (#600)
* improved object icons

* remove redundant control flow

* added support for base icon overrides by extension
+ deleted old icons folder and moved official icons into there

* fixed folder icon error

* removed temp nodes example

---------

Co-authored-by: Brendan Allan <brendonovich@outlook.com>
2023-03-09 11:08:10 +00:00
Oscar Beaumont d21a521cbe
[ENG-407] Spacedrop backend (#598)
* format Rust

* Spacedrop a string

* Praise thee Clippy, lord of the Rust

* add protobuf to Mac and Linux CI

* plz GH Actions have Chocolatey
2023-03-09 11:27:45 +08:00
Oscar Beaumont f746530289
P2P Core (#597)
* implement space time

* I love it when my code works
2023-03-09 03:18:33 +08:00
Brendan Allan dad2072d7b
generate + use ModelSyncData enum (#592)
* generate + use ModelSyncData enum

* cache prisma clients on sync-generator
2023-03-04 22:12:37 -08:00
Brendan Allan 29a3123df5
restrict all swift stuff to macos only (#590)
restrict all swift stuff to macos only
2023-03-03 21:29:52 -08:00
Brendan Allan d89240d9e2
Colocate native code + reorganise (#589)
colocate native code + all rust in crates/
2023-03-03 09:22:22 +00:00
Ericson "Fogo" Soares fe215566d6
Fix unique constraint on file path table (#585)
* Complying with a pedantic Clippy

* Some task runners for vscode

* Solving duplicated directories events
When creating a directory through MacOS's Finder, for some reason
FSEvents receives 2 Create Folder events that we have to handle

* Damn rustfmt
2023-03-02 02:36:49 +00:00
Jamie Pine 3f44d6f521
[ENG-363] Spacedrop UI + Misc Improvements (#568)
* begin spacedrop ui + misc ui improvements

* better 404 xox

* Update extensions.rs

I think I prefer Container

* added DragRegion component,  ot tested cuz im on my fone

* Update DragRegion.tsx

fix import

* added dummy drop items

* better dummy data

* added clouds & search bar

* added action buttons to spacedrop items

* customize subtle button

* added support for apng, thanks luka big pants

* use relative path in sidebar

* use BYTES const

---------

Co-authored-by: Brendan Allan <brendonovich@outlook.com>
2023-02-24 08:12:21 +00:00
jake 4cc582e08c
[ENG-134] OSS Licenses and Projects (#575)
* add `deps-generator` crate

* move `clap` types to separate file

* idiomatic filtering

* experimentally update ci

* add *really* basic dependency page

* remove `license_id` from `License`

* add bare minimum JSON files

* Revert "experimentally update ci"

This reverts commit c04897d902.

* re-insert comments

* Using reqwest blocking feature to avoid asyncness

---------

Co-authored-by: Ericson Soares <ericson.ds999@gmail.com>
2023-02-20 05:58:55 +00:00
jake c7dbc784cd
[ENG-361] Crypto crate docs and tests (#572)
* add hashing tests

* add encryption/decryption tests

* remove `reason`

* add file header tests (preview media deserialization is broken)

* fix keyslot reading bug

* add sd-crypto testing to ci

* add tests/constants for all hashing algorthms and param levels

* add blake3-kdf tests

* use `const` arrays for storing expected output

* test for `5MiB` `encrypt_streams` and `decrypt_streams`

* add invalid/mismatched nonce tests

* update `primitives` docs

* remove erroneous `,`

* grammar tweaks

* add errors to `#[should_panic]`

* cleanup `stream` tests

* cleanup hashing tests a little

* function docs
2023-02-16 11:42:30 +00:00
Utku b856f15b22
Tailwind Prettier Plugin (#557)
* add tailwind prettier plugin + run prettier

* add eslint tailwind plugin and eslint fix to turbo

* Ran eslint fix + fixed every other lint issues

* twStyle and lint stuff

* remove auto lint as it's buggy

* update eslint & add twStyle to tailwind regex

* Fix auto lint + config inconsistencies

* Add tailwind config to prettier config

* run lint:fix

---------

Co-authored-by: Brendan Allan <brendonovich@outlook.com>
2023-02-16 07:04:19 +00:00
Oscar Beaumont a9fceae819
improvements to spacedrive:// custom URI protocol (#550)
* fix `spacedrive://` custom protocol on Windows
(hopefully)

* custom protocol using `http::Response` + fix broken web

* import patches before App on web

* use `http::Request` for input to `handle_custom_uri`

* break  into dedicated file + error handling

* serving files via custom protocol

* cargo fmt because vscode did cringe

* lru cache to reduce video chunk request time

* add  helper to JS

* clippy be like

* remove duplicate Open buttons in context menu

* fix Linux 🙏

* no shot

* fix Windows custom URI passing (hopefully)

* better fix for custom uri on Linux

* upgrade Tauri for  feature

* switch url replacement order

* prevent React dev tools script being added in prod to desktop

* remove React devtools from html

* upgrade Tauri; required upgrading rspc, Axum, PCR

* pass typecheck + less cringe bigint

* clippy is love, clippy is life

* Typecheck plz

* fix bigint to number conversion

* use httpz + localhost server for Linux

* clippy be right

* Remove console.log

* [wip] proper auth

* fix Linux sidebar padding

* Secure Axum server with random

* Extracting app setup specific to linux to a different file

* remove outdated comment

* Some tweaks on cursom_uri.rs

* file_path_with_location doesn't need to be a named include

* fix typo

* factually wrong comment

* Change `unwrap` to `expect`

* bruh

---------

Co-authored-by: Ericson Soares <ericson.ds999@gmail.com>
2023-02-14 13:27:11 +08:00
Jamie Pine f47a2d58e5
[ENG 239] Onboarding Flow & Location Settings (#529)
* begin better onboarding

* added input and altered text

* better router & text + database icon

Co-authored-by: maxichrome <maxichrome@users.noreply.github.com>

* work on privacy screen + radio buttons

* fix video extension bug and alter screens

* add pending schema and location manager helper

* functional onboarding

* added secure temp store and started creating library loading screen

* fix secure temp keystore + api

* better onboarding

* added location settings and some overview concept, all WIP

* fix switch

* prep

* fix location router

* added backend settings

* attempted to fix form

* begin indexer rules editor, plus tweaks

* indexer rules coming soon

* fix onboarding img size

* cleanup

* clone is needed here, but clippy no like

* sike

* whole bunch of fixes

* clippy + ts

* Removing some TODOs from api/libraries.rs and fixing db size calculation

* moved object kind to client, added half functionality for appearance settings

* fix RadioGroup helper

* fix type issues

* cargo fmt

* fix creating library error handling + invalidate location list on update

* forgot to switch back to onError

* Invalidating getStatistics query on library creation and introducing the concept of waiting for a job on FileCopierJob

* F* cargo fmt

* fix RadioGroup interactivity

* wipe all migrations

* put back COLLATE NOCASE on extension columns

* update core.ts

* remove unused device component

* fix typeerror in mobile

---------

Co-authored-by: maxichrome <maxichrome@users.noreply.github.com>
Co-authored-by: Brendan Allan <brendonovich@outlook.com>
Co-authored-by: Ericson Soares <ericson.ds999@gmail.com>
Co-authored-by: Utku Bakir <74243531+utkubakir@users.noreply.github.com>
2023-02-10 14:08:13 -08:00
jake 5b78bbbb7d
[ENG-358] Malicious client protection for the KM (#562)
* add `queue_check`, `mount_check` and `unlock_check`

* make `unlock_check` marginally more readable

* clippy

* clean up & rename functions

* remove unlock check on `keys.list`

* use american spelling of `favorite`

* use `then_some` (thanks clippy)

* more american spelling
2023-02-09 09:30:26 +00:00
jake d1b6263ae7
[ENG-355] Keychain integration (and some typesafety) (#558)
* update crypto MSRV

* rename `keychain` to `keyring`

* make a start on the keymanager unlock refactor/keychain integration

* update routes

* update bindings

* add const identifiers

* add UI/front-end support for unlocking KM with OS keychains

* remove SK from lib creation dialog

* update query name

* add keyring functions

* attempt to update `change_master_password()` to use the keychain

* cleanup, fix master password change ui, better secret key in keyring detection

* cleanup TS a little

* add route for getting secret key from keyring

* update bindings

* update var names + show secret key in keys settings

* add `react-qr-code` and option to view the secret key (if it's in the OS keyring)

* allow copying SK to clipboard

* add `key_type` so we're not reliant on specific UUIDs for root/verification key handling

* clippy

* fix mobile typecheck

* fix typecheck, fix typo and tweak balloon hash parameters

* minor cleanup + typo fix

* use newtype structs

* WIP type refactoring (major readability boost!)

* update `use`

* add tokio `sync` feature

* too many structs? idk

* more cleanup

* add `generate` and `Nonce`

* `Nonce` and `Key` typesafety (beautiful)

* clippy + cleanup

* update code & examples

* fix bug & remove `ProtectedVec` as it looked out of place

* use `Key`

* add a query invalidation to make the UI extremely responsive

* ci pls work

* remove `keyringHasSk` route
2023-02-07 12:03:12 +00:00
Brendan Allan d1aa927203
Merge pull request #553 from spacedriveapp/non-unique-objects
Store CAS ids on file paths
2023-02-01 12:11:03 +08:00
jake 9fb01fc9d2
[ENG-341] OS Keychains (#554)
* super barebones keychain start

* working+unfinished linux keychain

* add `copy` to `Identifier` and add `delete()`

* add generic errors + code cleanup

* cleanup code & add support for apple keychains

* remove `users` dep

* use uppercase UUID and tweak apple `account` variable

* revert uppercase change and clean up linux labels

* code cleanup & add a readme to the crypto crate

* remove useless `map`

* correctly handle keyring not supported errors

* add `Send` to `Box<dyn Keyring>`

* remove useless `cfg`s

* only return `Ok()` if we support keychains (pls work CI)

* clippy

* Minor fixes

---------

Co-authored-by: Ericson Fogo Soares <ericson.ds999@gmail.com>
2023-02-01 00:11:03 +00:00
Ericson Fogo Soares dd1db0338e A bunch of minor fixes 2023-01-31 20:02:23 -03:00
Brendan Allan 494e87d115 Merge branch 'main' into non-unique-objects 2023-01-30 12:11:35 +08:00
Brendan Allan bf362eadd6 store cas ids on file paths 2023-01-28 11:42:24 +08:00
brxken128 6874c23c01
make requested changes 2023-01-26 10:39:18 +00:00
brxken128 2abf75defd
update things to use workspace tokio 2023-01-25 22:03:10 +00:00
brxken128 3f2a62feb4
Merge branch 'main' into eng-348-async-crypto 2023-01-25 22:02:23 +00:00
brxken128 a11d81549f
use a tokio::sync::Mutex just to be safe 2023-01-25 18:20:29 +00:00
brxken128 a3b1c8f5b8
tell everything to use workspace tokio + add tokio git patch 2023-01-25 14:36:29 +00:00
brxken128 7623bf9a6c
change fs and macros tokio features to dev-deps 2023-01-25 14:03:00 +00:00
brxken128 fa147f3e87
add correct features to tokio dep 2023-01-25 12:47:26 +00:00
Brendan Allan 3c7c05cc87 update shared sync functions to use new system 2023-01-25 13:17:01 +08:00
brxken128 2381798790
fix stream reading due to read()
Co-authored-by: Ericson Fogo Soares <ericson.ds999@gmail.com>
2023-01-24 23:28:16 +00:00
brxken128 c75e983fd4
bring in Salt, Key and EncryptedKey type aliases 2023-01-24 17:01:50 +00:00
brxken128 546bf643b2
add ProtectedVec type alias 2023-01-24 16:47:41 +00:00
brxken128 3ed106e27b
clippy 2023-01-24 16:35:31 +00:00
brxken128 3fd547ab2d
disable benchmarks as they need async support 2023-01-24 14:25:42 +00:00
brxken128 f13c247aec
update examples 2023-01-24 14:25:19 +00:00
brxken128 371ac93bcf
async crypto! 2023-01-24 14:19:27 +00:00
brxken128 87b75cfe62
return to match/for statements for async iterators (until rust gets better support) 2023-01-24 14:04:18 +00:00
brxken128 b929d36c89
more async crypto 2023-01-24 13:50:46 +00:00
brxken128 b077e16f00
mostly async keymanager 2023-01-24 13:16:47 +00:00
brxken128 64d92c2132
async stream encryption and decryption 2023-01-24 12:49:54 +00:00
brxken128 2d2f45be3d
Merge branch 'main' into eng-345-cut-copy-duplicate-actions 2023-01-23 11:03:27 +00:00
Ericson Fogo Soares 5e30cf6354 Removing some unnecessary clones and other small fixes 2023-01-20 18:22:17 -03:00
Brendan Allan e4e80fa603
Refactor routes (#535)
* refactor routes

* use default export for screens

* use "route" relative routing

* don't use absolute import for root screens

* no absolute importing folders!!!

* fine we can absolute import folders
2023-01-20 12:40:36 -08:00
Brendan Allan 24de617b92
Fix webkit reload (#534)
* wrap WKWebView in UnsafePointer

* formatting

* more formatting
2023-01-20 00:25:12 -08:00
maxichrome ec4352a30a cleanup(don't hate me): prettier format everything 2023-01-19 19:32:35 -08:00
jake 0a31e7f8ce
[ENG-257] File erasure and deletion (#524)
* add helper functions for getting context-menu related filesystem info

* update comments and such

* add a barebones delete job

* add file deletion route

* add (empty but working) delete file dialog

* add super experimental file erase function

* update function doc

* add erase files route

* add erase files job

* tweak file erasing in crypto crate

* bindings

* add erase files dialog

* add missing `?`

* fix erase file form trickery

* remove unused imports

* terrible attempt at recursion for file erase

* IT WORKS

* code cleanup + clippy

* async just dropped (not for erase yet)

* async erase just dropped (+async crypto crate)

* add "nice data" for jobs
2023-01-18 19:55:14 -08:00
jake 0ef7f64bbb
[ENG-328] Keymanager mounting queue (#514)
* add `get_queue` and `is_queued` to KM and router

* mark keys as being mounted/finished mounting, use a `DashSet` instead of vec

also includes temporary `dbg!()`s for manual valdiation that it works

* remove dbg and fix build (works great)

* mark key manager as unlocking correctly

* cleanup keys api

* add `KeyAlreadyQueued` error

* use react mounting queue

* fully prevent against multiple mount jobs of the same key with the rust queue

* show keymanager as unlocking in the UI

* handle query invalidation for `isKeyManagerUnlocking` with a closure (semi-temporary)

* revert changing text (doesn't work where the keymanager is unlocked from, only the other unlock screen)

* clippy

* add `useRef` for mounting queue within react

* pass `ref.current` to `Key` instead
2023-01-17 06:19:54 -08:00
Brendan Allan 8e479f736f
Eng 332 auto generate sync ids (#523)
* generate sync ids and use model name from ModelActions

* cleanup AddLocationDialog imports

* rustfmt
2023-01-15 07:25:11 -08:00
Brendan Allan ae94ada4f8
Begin form abstraction (#515) 2023-01-12 07:26:59 -08:00
jake 0d554e9d06
[ENG-331] StoredKey overhaul (#513)
* add wip storedkey versioning

* storedkey versioning! (not pretty, but it never will be)

* add version to `StoredKey` and re-gen migrations to handle serde

* use `serde` for interacting with the DB + handle errors
2023-01-11 05:15:08 -08:00
jake 4536710586
Detect SD-encrypted files and display them as such (#511)
* add encrypted file magic bytes/extension

* use archive icon for encrypted files temporarily

* use pascal case for encrypted extensions

Co-authored-by: Brendan Allan <brendonovich@outlook.com>
2023-01-11 00:40:49 -08:00
jake fbd8b6d199
[ENG-327] Crypto refactor (#510)
* fix secret key showing as `Some()` when it's empty

* rename function

* use `uuid` feature of `rspc`  so clippy works with `--all-features`

* remove dead error

* update clippy rules

* more idiomatic + cleaner hashing, that handles secrets better

* remove dead code + minor refactor of the KM

* cleanup`KeyManager::get_key()`

* cleanup keys router

* variable inlining, idiomatic rust, etc

* clean up encrypt/decrypt jobs

* embed preview media into encrypted files

* even more idiomatic rust

* fix previous file extension code (no amount of idiomatic rust can fix bad patterns)

* hopefully make large serialization slightly more efficient

* remove dead code + old tests

* inline key derivation (not so sure about this one)

* use `then` instead of `if`

* stop using expensive clones

* add `Protected::into_inner()` to prevent `.expose().clone()` patterns

* use `tokio::fs::File` where possible

* add and use `JobError::MissingData`

* use `|| null` for the secret key

* use `block_in_place` for large r/w sync tasks
2023-01-10 12:36:28 -08:00
jake c1b2b1f5ee
[ENG-323] Key manager onboarding fixes (#506)
* add hash with secret functions

* refactor KM to make the secret key both optional, and user-settable

* update library create function and create library dialog

* fix annoying MPC typo

* fix ordering of key manager creation and seeding

* overhaul master password changing UI to support the potentially empty secret key

* patch keymanager unlocking for potentially empty SK

* make create library dialog text more fitting

* remove dead code

* add autogeneration, view/hide, copy to clipboard icons

* cleanup older code

* make onboarding screen look a tad better

* minor UI tweaks

* patch backup restoration UI

* add pw generation to MP change dialog

* make sure passwords are same before submitting lib creation

* UI fixes and allow setting algorithm/hashing algorithm for master password

* make sure KM is re-seeded on SD load and prevent loading duplicate keys into the keystore

* remove old comment

* small organization

* update encryptfiledialog

* remove dead KM code

* cleanup some TS

* move `PasswordMeter` to separate file

* refactor master password change dialog

* cleanup the TS and improve string -> hashingalgorithm conversions

* add `key` values to selectoption

* capitalise `BLAKE3`

* fix sorting for `getHashingAlgorithmString`

* remove onboarding route

* Revert "add `key` values to selectoption"

This reverts commit 10e6456f22.

* add `key` to `SelectOptionKeyList`

* fix using rspc errors directly

* Update packages/interface/src/components/onboarding/Onboarding.tsx

Co-authored-by: Brendan Allan <brendonovich@outlook.com>

* use `navigator.clipboard.writeText`

* move `OnboardingConfig` to crypto crate

* move `zxcvbn` options outside of component

* further cleanup

* fix mobile typecheck

* fix keylist typo

* improve `SelectOptionKeyList` further

Co-authored-by: Utku Bakir <74243531+utkubakir@users.noreply.github.com>
Co-authored-by: Brendan Allan <brendonovich@outlook.com>
2023-01-06 17:03:22 +00:00
Brendan Allan 676e512c4a
Beginning of sync stuff (#502) 2023-01-04 23:11:55 -08:00
jake 22bc77a39e
[ENG-319] Balloon hashing (#489)
* add key attribute to `Key` in `ListOfKeys`

* add balloon hashing function with untailored parameters

* add ser/de rules for blake3-balloon (and change argon2id's)

* fix benchmark

* use `to_bytes`, `from_bytes` and `from_reader`

* cleanup code

* add blake3-balloon options to the UI and fix library sync/automount enable bug

* cleanup some serialization code

* fix hashing algorithm deserialization

* clean up header serialization + more idiomatic master key decryption

* clippy

* add generic ser/de error to crypto crate

* fix `Display` and crypto cli

* move crypto cli to cli app

Co-authored-by: Brendan Allan <brendonovich@outlook.com>
2023-01-04 09:57:02 +00:00
jake 4569e45e47
[ENG-312] Crypto details CLI tool (#500)
* add untested crypto details tool

* clean up output a little

* clippy

* update crate details and `clap`
2023-01-04 00:15:12 +08:00
Oscar Beaumont 3964d44ce5
Integrate KeyManager with library creation flow (#491)
* integrate keymanager with library creation flow

* final fixes

* fix clippy recommendations

* minor fixes on library create dialog

* reenable `panic!` for no key + fix secret handling code

* prevent user setting secret , instead hardcode it

* clean library manager default key selection

* bring back wrongly removed `keys.onboarding` resolver

* fix types in `CreateLibraryDialog`
2023-01-04 00:13:12 +08:00
jake db5b401238
[ENG-318] Implement a KDF to derive new keys (#487)
* add blake3, `derive_key()` and clean up code

* fix a couple of things from the previous commit

* add context strings for root/file key derivation

* add salt to schema

* update refs of `salt` to `content_salt` within the keyslot

* cleanup code and add kdf salt to the keyslot

* rename salt to content salt in examples

* cleanup header code + remove dead code

* implement key derivation for keyslots

* gen new migrations that contain a salt column

* keymanager refactor (code is very idiomatic now) - needs thorough testing

* further cleanup

* clippy

* add a master password context string

* use key derivation for deriving keys from the root key

* update to use new code and remove `match` from `en/decrypt_bytes()`

* clippy

* use less unwraps in library manager code

Co-authored-by: Oscar Beaumont <oscar@otbeaumont.me>
2022-12-30 20:09:44 +00:00
Ericson "Fogo" Soares bcbcd260d4
ENG 223 Location Awareness (#468)
* Introducing location online/offline checks and location relink

* Some initial drafts for location watchers

* Location metadata relink and add library

* Many improvements at job system
Now using prisma batching at identifier job
Removing blocking I/O from extension subcrate
Implementing lazy generation of thumbnails
New current directory identifier job to be used on light rescans

* Some optimizations on identifier and object validator jobs

* merge jamie's identifier PR

* fully repaired identifier job

* properly hooked up object kind

* inspector fix

* fix video badge

* small improvements to libraries settings

* identifier and inspector improvements

* fix feature flags and hook up context menu location utilities

* BETTER CONTEXT MENU x100

* test-files

* style tweaks

* new icon designs

* manifest

* fix thumbnails on web

* media data

* New Location Watcher and some minor fixes

* disable broken media_data extractor, wip

* wip

* function name fix

* Fixing pnpm prep and some warnings

* Solving a race condition beetween indexer job and FS event handlerSome other minor warnings

* Generating thumbnails on watcher

* Remove event handler on watcher

* Some initial works on modify events and other small fixes

* File update event

* Trying to be more generic with used events and some tests to validate our assumptions

* Turning on location metadata file

* Introducing core unit tests on CI pipeline

* Submiting new unit test assumptions to validate on windows CI

* Fixing unit tests

* Fixing unit tests again

* Fixing unit tests

* Fixing unit tests for macos

* Fixing unit tests for macos again

* New structure for platform dependent event handling
Implementing event handlers for Linux and MacOS

* minor fixes + rustfmt + clippy

* Windows event handling

* Introducing a feature gate to only use location watching on desktop app for now

* Putting more stuff behind feature gates to avoid warnings

* Adding feature to cargo test on CI

* Changing some debug logs to trace logs and removing Jamie specific stuff

* Make location removal from manager less async

* fix build when "location-watcher" feature disabled

* fix types + clippy

* make location manager non-static

* remove uses of `to_string_lossy`

* more invalidate_query calls

* Expose `library_ctx` directly to avoid needless clones

* New materialized_path handling for directories

* Removing cascade delete between file_path and object
- Some other minor stuff

* remove unused `CurrentDirFileIdentifierJob`

Co-authored-by: Jamie Pine <ijamespine@me.com>
Co-authored-by: Oscar Beaumont <oscar@otbeaumont.me>
2022-12-31 00:53:24 +08:00
jake bbca699e33
[ENG-316] Decrypt with Password (#484)
* update generated key on slider click

* add ui for decrypting with password+saving to key manager

* update rust code to support password decryption and saving it to the library

* add decrypt with password UI functionality

* add content salt to key viewer

* disable key manager decryption if no keys are mounted

* clippy
2022-12-21 13:04:52 +08:00
jake 78a5754f4e
[ENG-315] Key viewer/general improvements (#483)
* add encryption+hashing algorithm to key viewer dialog

* abstract `save()` and `open()` with `Platform`

* add automount to `StoredKey` and offer option to enable/disable it for keys

* fix missing statement exec

* general tidy up

* add sync to library button and cleanup default setting code

* use alert dialogs for unsupported `open`/`save` operations (web)

* tweak alert message
2022-12-13 09:48:14 +08:00
jake 8ee2d18053
[ENG-309] Envelope encryption for the KM's root key (#482)
* use master key to encrypt the root key (breaking)

* introduce breaking changes to the keymanager (this breaks master password changing, backup restoration, etc)

* change MP -> RK

* minor tweaks + comment out broken features temporarily

* experimental new master password changing

* add a `write_storedkey_to_db` helper function

* general cleanup and fixes

* potentially patch backup restoration

* uncomment+clean up restoration in api

* strip whitespaces from user inputted secret key, and fix MPC bug

* clippy

* add memory only flag to `StoredKey`
2022-12-12 15:04:08 +00:00
jake f1292d6e51
Crypto Hotfixes (#477)
* fix annoying typo that was breaking master password changing under certain conditions

* move `Select` boxes out of `react-hook-form` so they work

* add `DummyKey` and rework the UI a little

* fix bug where user could select a hashing algorithm (it did nothing)

* fix bug where encryption algorithm would be inherited from the key, not the user's choice
2022-12-09 22:05:32 +08:00
jake 0eb0376d5b
[ENG-308] Crypto optimizations (#470)
* add benchmarks for encryption/hashing

* add more benchmarking sizes

* general performance improvements

* massive performance gain with minimal security impact

* update `sd-crypto` author info
2022-12-08 12:08:35 +00:00
jake 70b063b002
[ENG-311] Hex secret key encoding (#476) 2022-12-07 15:17:14 +00:00
jake 5651ac3f69
[ENG-251] Crypto Jobs (#475)
* add base crypto job

* leave hashing to the KM and not keyslots

* add latest header object versions and update examples

* add a very experimental (local file!) encrypt job

* working file encryption with default key

* working file decryption

* non-working encrypt file dialog

* general code cleanup

* comments and correct feature locking

* fix the `Dialog`

Co-authored-by: maxichrome <maxichrome@users.noreply.github.com>

* WIP encrypt dialog

* massive encrypt dialog overhaul

* add generic explorer alert dialog

* better key naming/numbering, alerts for unsupported scenarios

* cleanup + decrypt file dialog

* remove debugging stuff

* remove dead code & fix a bug with key select being empty

* clean up dialog UI massively

* add support for setting output path

* add correct button type to prevent early dialog submits

* add "nice" job data for encrypt/decrypt

* tweak functionality of key setting in encrypt dialog

* include full file metadata if selected, and lay out groundwork for preview media inclusion

* add comment about not restoring metadata/pvm just yet

* general cleanup

Co-authored-by: maxichrome <maxichrome@users.noreply.github.com>
2022-12-07 15:16:13 +00:00
brxken128 bc4e8cb9c8
add clippy allow so Protected can work correctly 2022-12-07 15:04:57 +00:00
brxken128 4449bcb143
code cleanup and use hex for secret key encoding 2022-12-07 15:00:14 +00:00
jake 2baf16d982
[ENG-307] Key manager features (#467)
* working key management in settings page

* sync with library button

* fix `rspc` feature and add passphrase generation

* untested key manager rework

* trying to return values from mutations

* update library manager and remove settiong master PW

* update bindings

* set static secret key/master password

* prompt user for master password if correct one hasn't been provided yet

* add `hasMasterPassword` route

* add `clearMasterPassword` route + remove dead code

* tweak `set_master_password()` and add dedicated error

* tweak UI, fix `few hooks than expected`, add unmount+lock button

* remove old comment

* fmt

* clippy

* move static key/password setting so it doesn't fail sometimes

* add dedicated `get_key()` and remove keys from memory

* add `getKey` route

* update bindings

* use `const` instead of `let`

* comment updates

* update schema to remove salt

* add string parse error

* generate passphrase within key manager

* generate new migrations

* feature gate serde support in `crypto` crate

* fmt

* more specific error types

* foramatting

* add locking mechanism to keysettings page (not working?)

* fix react hook issues

Co-authored-by: maxichrome <maxichrome@users.noreply.github.com>

* remove empty onclick

* add keymanager dropdown menu

* working key backup functionality

* add experimental master password changing support

* update bindings

* semi-working change master password dialog

* use gear/lock icons to clean up key manager UI

* make settings button functional

* make buttons uniform and format code

* fix double base64 encode

* add change master password dialog and secret key dialog

* code cleanup

* restore backup dialog

* change UI wording

* make a start on restoring from a backup

* potentially working keystore restore

* don't overwrite verification key if one is set

* working backup restore + fix master password changing

* fix typo in static password/verification key check logic

* change wording to make UI clearer

* disable mount button if key is empty

* handle errors+remove type annotations

* show total imported keys on backup restoration

* add zxcvbn package

* change input border colour based on zxcvbn score

* clippy and formatting

* password strength meter

* remove nbsp

* add button type to stop early form submission

* use `react-hook-form` for backup restoration dialog

* more `react-hook-form` stuff

* attempt to fix password meter

* small cleanup

* Fix password meter

* update colours

Co-authored-by: maxichrome <maxichrome@users.noreply.github.com>
Co-authored-by: Utku Bakir <74243531+utkubakir@users.noreply.github.com>
2022-12-02 10:18:21 +00:00
jake a403224b3b
[ENG-262] Key Manager Integration (#450)
* add keys router

* make progress on keymanager integration

* make name non-negotiable

* hyphenate encryption algorithm names

* Revert "make name non-negotiable"

This reverts commit 9c0f51329e.

* add some more keymanager queries

* add master password and default key routes

* add newly registered key to db + fmt

* clippy, formatting and `updateKeyName` route

* add automount to schema+automount keys where `true`

* update bindings

* working key add/mount

* working keylist

* mounted keys show first

* cleanup code

* add comments, code cleanup, more functions

* unmount all keys button

* comment and keymanager `clear_master_password()`

* add no keys available message

* fix unmount button

* use dashmap for concurrency

* fix missing keylist issue and add invalidate query macro

* set correct RSPC types

* statically set master password (TEMPORARILY)

* add remove key function within the keymanager

* key dropdown menu and impl

* formatting

* allow `option_if_let_else`

* add comment about key stats

* add additional comment

* rpsc error handling for the keys route

* fix rspc errors with an impl

* crypto crate errors to `sd-crypto::Error`

* remove `map_err`

* use custom result type

* cargo fmt

* clippy

* fix builds

* remove `Error::MutexLock`

* fix unnecessary unwrap

* mutex error handling (buggy for some reason)

* clean default key logic

* fix default key clearing

* allow a key to be removed without bugs

* implement requested changes

* use a single `useMemo`

* update schema with defaults

* re-generate migrations

* use rust enums in TS

* remove dead code

* remove mutate expansion

* read key list from keymanager, not prisma

* add "Default" key marker and cleanup TS

* rustfmt

* remove dead code
2022-11-05 04:18:01 -07:00
jake 92a3de4973
Fix metadata/preview media sizes for 32-bit targets (#446)
replace `usize` with `u64` for md/pvm lengths

Co-authored-by: Utku Bakir <74243531+utkubakir@users.noreply.github.com>
2022-10-29 02:07:10 -07:00
Brendan Allan b713c896a0
More Sync Stuff (#438)
* broken sync example

* move to react

* filter owned ops more

* update deps

* working sync

* relation operations

* fix pnpm lock

* please rustfmt

* fix api

* compare strings properly

* please clippy

* feature gate logging

* use bleeding rspc in example

* use bleeding rspc

* use generated rspc client
2022-10-27 23:12:28 -07:00
maxichrome 9d72de352b
Improve setup script (#436)
* Improve bash setup script (less silent!)

* use proper FFmpeg capitalization in relevant docs

* Fix typo in Homebrew error message

Co-authored-by: maxichrome <maxichrome@users.noreply.github.com>
2022-10-25 18:08:50 -07:00
Jamie Pine 3a3ec6251f Merge branch 'main'
Conflicts:
	packages/interface/src/screens/settings/library/LibraryGeneralSettings.tsx
2022-10-23 18:43:27 -07:00
jake 0c7aed5f86
[ENG-258] Key Manager and Crypto Crate Improvements (#423)
* add base keymanager structs/functions

* change md/pvm `new()` to prevent useless re-hashing

* update lockfile

* update keymanager

* cleanup code, make things easier to understand

* move md and pvm construction to the header

* update pvm ser/de

* update metadata ser/de

* additional API changes, update example

* formatting and clippy

* update examples

* move `impl` to associated files

* formatting+clippy

* add more keymanager functionality

* add docs

* wrap `master_password` as an option

* add `sd-crypto` crate as a dependency to `core`

* add key manager to `LibraryContext`

* rename `id` -> `uuid`

* add more keymanager functions

* add `set_master_password()` function

* function to see if keymanager has master password

* update schema

* add default bool to schema

* populate keystore on SD startup

* clippy+formatting

* implement requested changes

* remove unwrap

* fmt, clippy, remove ser/de derives

Co-authored-by: Brendan Allan <brendonovich@outlook.com>
2022-10-23 03:47:41 -07:00
Jamie Pine 6ecdc6a02f finish theming 2022-10-23 03:24:46 -07:00
Jamie Pine c4226f0e7a Merge branch 'eng-259-identify-objectkind'
Conflicts:
	apps/landing/package.json
	apps/landing/vite.config.ts
	packages/interface/src/AppLayout.tsx
	packages/interface/src/components/explorer/Explorer.tsx
	packages/interface/src/components/explorer/ExplorerTopBar.tsx
	packages/ui/src/ContextMenu.tsx
	pnpm-lock.yaml
2022-10-20 22:49:15 -07:00
Brendan Allan 22330e6e71 sync docs 2022-10-19 14:44:09 +08:00
Jamie Pine 4acb8cc594
Merge branch 'main' of https://github.com/spacedriveapp/spacedrive 2022-10-17 01:36:05 -07:00
jake b5c571541e
[ENG-256] Crypto Crate Refinements (#410)
* add experimental preview media support

* clippy+formatting

* add support for mk/pvm nonces

* fix cursor positioning+size deserialization bug

* clippy+formatting

* cleanup code and add `decrypt_preview_media()`

* update comment

* remove `Mode` from `Keyslot`

* use little-endian `usize` for media length

* remove memory objects

* remove mode enum

* add `.zeroize()` to `Protected<>`

* remove `Mode` ser/de

* refactor, remove `Mode`, add helper functions

* formatting

* remove unused error

* update comments/docs

* comments, fix large files, insane performance

* doc updates and fix decrypt zeroize

* revert to 1048576 byte `BLOCK_SIZE`

* `Keyslot` and `PreviewMedia` constructors now handle encryption

* add metadata item with associated functions

* update comments, clean up useless items

* add metadata support within the header

* remove stray 128 in length calcs

* include metadata in header ser/de

* fmt+clippy

* rework keyslot decryption

* formatting

* api changes, code cleanup

* docs for `stream.rs`

* massive documentation re-write

* mark expensive doc test as ignore

* minor api tweaks

* add examples

* formatting+linting

* finalise documentation and add more examples

* formatting
2022-10-13 15:25:42 -07:00
Jamie Pine b94d2ed736
(hotfix) p2p preventing compile 2022-10-08 11:30:17 -07:00
jake d236b28f84
[ENG-250] Crypto Library (#400)
* add crypto crate with some functionality

* formatting

* add `argon2id` parameter levels

* add descriptive comments

* add stream decryption objects

* add `StreamEncryptor` struct

* add `StreamDecryptor`

* general cleanup

* add `thiserror` and error handling

* add header structs

* add basic serialization functionality

* advance serialization

* finish serialization

* clean up serialization and use `impl`

* finalise deserialization

* add stream helper functions and remove old code

* add AAD creation and retrieval

* add important comment

* add `ChaCha20Rng` as a CSPRNG

* cleanup and crate-wide clippy lints

* apply nursery lints

* add in-memory encryption objects

* rename `utils` to `objects`

* move (de)serialization rules to separate file

* add header-write helper function

* add password hash helper function

* add `decrypt_master_key` function

* cleanup, formatting, linting

* move keyslots to separate file, and rename them

* add basic comments

* remove `secrecy` dependency and import `protected`

* add `to_array` helper function

* `sd_crypto` -> `sd-crypto`

* remove manual drops

* add clippy allows

* add `new()` for `Keyslot` and `FileHeader`

* remove license

* zeroize read buffer on error

* magic bytes are now `ballapp`

Co-authored-by: Brendan Allan <brendonovich@outlook.com>
Co-authored-by: Jamie Pine <32987599+jamiepine@users.noreply.github.com>
2022-10-07 07:31:40 -07:00
Jamie Pine 900ee618ce
Merge branch 'main' of https://github.com/spacedriveapp/spacedrive 2022-10-06 18:10:08 -07:00
Jamie Pine e402fc52b7
sd extensions crate
Co-authored-by: Brendan Allan <brendonovich@outlook.com>
2022-10-06 18:10:03 -07:00
Oscar Beaumont aacdf33216 merge duplicate prettier config and format repo with it 2022-10-07 07:45:15 +08:00
Oscar Beaumont fe4da9048a cargo fmt + fix clippy lints 2022-10-07 07:39:59 +08:00
Oscar Beaumont 75c4680b43 Merge remote-tracking branch 'origin/main' into merge-p2p-code 2022-10-07 05:56:05 +08:00
Oscar Beaumont d7fddc83f7 fix major bugs in p2p PR 2022-10-07 05:51:40 +08:00
Oscar Beaumont 598ba38a9d merge in P2P code 2022-10-07 05:32:45 +08:00
Oscar Beaumont 20523a94c9 upgrade workspace dependencies 2022-10-06 14:03:58 +08:00
Brendan Allan 723058e060 merge main into update-rspc-and-more 2022-10-05 08:58:03 +08:00
Brendan Allan ed06e3051e
[ENG-84] Sync library (#394)
* new sql lib stuff

* add sync stuff + rename all crates

* build sd-core

* add sync/example/dist to source

* fix sync example in monorepop

* appease clippy

* update lockfile

* update commit hooks

* fix typescript

* fix typescript build

* please rustfmt
2022-10-04 05:25:12 -07:00
Brendan Allan 66dd9aa28b
move prisma schema and migrations to /core (#392)
* move prisma schema and migrations to /core

* fix prisma updates
2022-10-04 02:32:46 -07:00
Oscar Beaumont 60e228ec94 upgrade rspc 2022-10-02 01:16:11 +08:00
Jamie Pine 3e8c33d1c0
Added crates folder (#389)
* restructure core for crates folder

- improve naming consistency
- added placeholder files for near future code
- added docs

* - removed all typescript from core
- moved rspc bindings to client

* remove useless git ignore & rustfmt
2022-09-29 21:02:29 -07:00