Commit graph

288 commits

Author SHA1 Message Date
Ericson "Fogo" Soares 68ca2382fd
[ENG-1627] Move all jobs and actors to an "Old" namespace (#2157)
* Bunch of new warns due to a stronger clippy config

* Moving old processing stuff to a new namespace
Also fixing a bunch of typos through the entire codebase.

* Rustfmt
2024-03-04 18:57:35 +00:00
Brendan Allan 54af5a822f
fix doAdd cloud api and provide operations count (#2154) 2024-03-01 10:41:47 +00:00
Oscar Beaumont 2cc828c4af
Everything over P2P (#2135)
* Fix websockets over P2P + multi-node cache

* cleanup features

* Explorer over P2P demo

* wip

* wip

* working for ephemeral stuff

* Button to get out of remote viewer

* remove

* Cleanup

* more cleanup

* Fix cross-node library queries

* Files over P2P

* Drop `Header::File`

* Fix Clippy

* Don't let noobs get hacked
2024-02-29 07:46:30 +00:00
Brendan Allan e8f329de49
[ENG-1614] Remove @default from synced models (#2134)
Remove @default from synced models
2024-02-28 04:56:29 +00:00
Ericson "Fogo" Soares dba85ebac3
[ENG-1513] Better integration between Jobs and processing Actors (#1974)
* First draft on new task system

* Removing save to disk from task system

* Bunch of concurrency issues

* Solving Future impl issue when pausing tasks

* Fix cancel and abort

* Bunch of fixes on pause, suspend, resume, cancel and abort
Also better error handling on task completion for the user

* New capabilities to return an output on a task

* Introducing a simple way to linear backoff on failed steal

* Sample actor where tasks can dispatch more tasks

* Rustfmt

* Steal test to make sure

* Stale deps cleanup

* Removing unused utils

* Initial lib docs

* Docs ok

* Memory cleanup on idle

---------

Co-authored-by: Vítor Vasconcellos <vasconcellos.dev@gmail.com>
2024-02-26 19:45:58 +00:00
Brendan Allan c533d12df0
media data sync (#2102)
* basic sync operation backfill

* media data sync

* sync entry helpers

* fix sync generator

* nicer

* re-add key_id
2024-02-21 11:42:10 +00:00
Oscar Beaumont 519b1b6c46
Fix P2P not working for libraries (#2031)
* P2P Debug route

* Remove legacy peer to peer pairing process

* Fix error typo

* Sync instances with cloud

* Upgrade deps + extended instance data

* Create instance with extended metadata

* Auto sync instances

* Actually `.await`

* bruh

* sync library info

* this isn't gonna work

* only sleep cloud receiver when no more messages (#1985)

* [ENG-1567] Fix renaming (#1986)

fix rename

* only sleep cloud receiver when no more messages

* use in memory instances during cloud receive (#1995)

* use in memory instances during cloud receive

* is_empty

---------

Co-authored-by: nikec <43032218+niikeec@users.noreply.github.com>

* fix type error

* wip

* make mdns mdns better

* rebuild state

* Add hooks + listeners + discovered state

* Split into crates

* wip fixing core + wip merging Spacetime into `sd-p2p2`

* `SmartLockGuard` + `Listener`

* Make `sd-core` compile

* Reenable all operation receivers

* Fix all broken code within `sd-core`

* minor fixes found in review

* Bring in `libp2p` + restructure `sd-p2p` for the gazillion-th time

* whoops

* Compile no matter the (runtime) cost

* fixing merge issues

* wip

* a

* b

* C

* Handle port betterer

* c

* Migrate node config

* a

* no crash on startup

* wip

* a

* jdfhskjfsg

* a

* fix discovery

* a bunch of fixes

* getting Spacedrop working

* I don't get why it no worky

* debug example

* a

* wip

* wip

* removing logging from stream impl

* wip: shit is fucked

* Redo quic integration  + Spacedrop working

* Fix shutdown - deadlocks + shutdown peers

* Add Prisma migrations

* Fix shutdown

* a

* fix

* cleanup

* The lord clippy hath spoken

* disable P2P settings for now

---------

Co-authored-by: Brendan Allan <brendonovich@outlook.com>
Co-authored-by: nikec <43032218+niikeec@users.noreply.github.com>
2024-02-21 08:13:40 +00:00
Utku d54782d7a2
Bump version to 0.2.4 (#2080)
* bump

* cargo

* bump height so it doesn't block onboarding

* disable parallelism for image labeler

---------

Co-authored-by: Vítor Vasconcellos <vasconcellos.dev@gmail.com>
2024-02-14 07:40:01 +00:00
Vítor Vasconcellos 72efcc9f62
[ENG-1595] Fix some problems with the AI system (#2063)
* Fix some problems with the AI system
 - Fix downloading model using an internal rust string representation as path for the model file
 - Fix Linux loading onnx shared lib from a hardcoded path
 - Fix App should not crash when the AI system fails to start
 - Fix sd-server failing to start due to onnxruntime incorrect linking
 - Some extra clippy auto fixes

* Use latest ort

* Fix dangling sd_ai reference
 - Use entrypoint.sh to initilize container

* Fix server Dockerfile
 - Fix cargo warning

* Workaround intro video breaking onboarding for the web version

* Fix rebase
2024-02-12 17:45:17 +00:00
Brendan Allan 177b2a23d6
sync support for labels (#2070)
* more sync support for file paths + saved searches

* sync support for labels

* update sync prisma generator to support more than tags

* workey

* don't do illegal db migration

* use name as label id in explorer
2024-02-09 13:20:51 +00:00
Brendan Allan 6f28d8ec28
More sync support for file paths + saved searches (#2067)
more sync support for file paths + saved searches
2024-02-09 08:17:04 +00:00
Oscar Beaumont a30deaa144
Cloud + P2P (#1970)
* P2P Debug route

* Remove legacy peer to peer pairing process

* Fix error typo

* Sync instances with cloud

* Upgrade deps + extended instance data

* Create instance with extended metadata

* Auto sync instances

* Actually `.await`

* bruh

* sync library info

* this isn't gonna work

* only sleep cloud receiver when no more messages (#1985)

* [ENG-1567] Fix renaming (#1986)

fix rename

* only sleep cloud receiver when no more messages

* use in memory instances during cloud receive (#1995)

* use in memory instances during cloud receive

* is_empty

---------

Co-authored-by: nikec <43032218+niikeec@users.noreply.github.com>

* fix type error

* wip

* make mdns mdns better

* response

* remove renames

---------

Co-authored-by: Brendan Allan <brendonovich@outlook.com>
Co-authored-by: nikec <43032218+niikeec@users.noreply.github.com>
2024-02-02 09:31:11 +00:00
Brendan Allan c45f6d7bfa
separate cloud sync into sd-core-cloud-sync (#2025)
* separate cloud sync into sd-core-cloud-sync

* ci
2024-01-31 08:16:54 +00:00
Vítor Vasconcellos 556ddbd4f8
Port AppImage build to use appimage-builder (#1785)
* Initial port to appimage-builder

* Almost

* Fix appimage build on arm64

* Custom patch for external binaries run under appimage
 - Disable bubblewrap sandbox when running under appimage
 - Change cwd to APPDIR when running under appimage

* AppImage Works (for the first lunch, then it crashes with Stack Smash 😭)

* Fix stack smashing, now AppImage almos fully works \o/ (gstreamer is still broken)
 - Temporarily disable the volume watcher when running under appimage (Workaround for the stack smash error)
 - Wrap gnu lic version check for appimage under conditional compile for glibc targets
 - Add error handling for the justUpdatedCheck
 - Fix VITE_LANDING_ORIGIN being undefined

* On non glibc systems default to runtime/compat

* Use glibc version 2.8 instead of 0 for non-gnu systems

* Fix video playback not working due to broken GstRegistry

* Build and publish new AppImage release artifact
 - Fix model location when building deb
 - Improve model path resolution logic
 - Remove patchelf dependency from setup script
 - Fix incorrectly ignore gstreamer dependency in AppImage recipe

* Fix clippy complaining about `get_path_relative_to_exe`
 - Read GLIBC_FAKE_VERSION or use 2.8 for musl in appimage (while the code is there, this is not really supported for now)

* Remove appimage tauri target from release CI

* Remove setup-buildx-action, not relly needed

* typo fix

* Fix git describe command running on cwd instead of the repo root dir

* Attempt fix weird git permissions errors in CI+docker

* Pass CI env to docker appimage

* Only use git after installing it

* Pass target to appimage build script

* Fix permission after creating appimage

* -_-

* Swap envvar with github ci var

* Format

* Add instruction on how to manually build an AppImage

* Fix typos

* docs: add note about running podman with `--privileged` if there's a permission denied error

* docs: fix typo and link directly to appimage-building `README.md`

* refactor: streamline code and make it a bit cleaner

---------

Co-authored-by: jake <77554505+brxken128@users.noreply.github.com>
2024-01-22 10:52:26 +00:00
Julian Braha 9018b9277f
Implement Hash, PartialEq, and Eq for CacheNode (#1964) 2024-01-21 01:28:06 +00:00
Julian Braha 04d6b2646a
Add missing zeroize to Identity (#1962)
Add derive(ZeroizeOnDrop) to Identity
2024-01-20 15:15:06 +00:00
Jamie Pine fdf31fc3a8
[ENG-1511] Library Screens (#1903)
* init

* changes

* Now updating statistics once a minute

* More robust statistics updater

* Concurrency is hard

* improvements to stats

* refactor

* adjust setting back/forward padding so it matches top bar

* refactor sidebar

* rename

* setting up screens

* some changes

* Co-authored-by: Brendan Allan <Brendonovich@users.noreply.github.com>

* yes

* yes2

* refactored explorerItem.ts

* important explorer code shouldn't be thrown away in a util moment

* support for multiple thumbnails in ExplorerItem

* clippy

* move debug

* yes

* label filters

* ts

* comment out unconnected stuff

* added .mid for midi files

---------

Co-authored-by: Ericson Fogo Soares <ericson.ds999@gmail.com>
Co-authored-by: Brendan Allan <brendonovich@outlook.com>
2024-01-16 12:15:03 +00:00
Brendan Allan 4962b1160b
[ENG-1488] Cloud sync message compression (#1922)
* CRDTOperationWithoutInstance

* almost there

* fully compress messages

* implement more of sd-cloud-api

* sd-cloud-api-ify

* landing -_-

* openssl

* clippy

* idk

* bruh

* wut

* ahhh right
2024-01-09 10:21:23 +00:00
Oscar Beaumont bef1ebcade
[ENG-1508] Launch Spacedrop (#1893)
* Shit UI

* refactor a bit

* wip

* yeet

* farming the wry but it's stale

* Real-time hover event

* Hook with `Platform` + fix broken window-state plugin

* `DragAndDropDebug`

* Clippy

* revert Tauri v2 stuff

* minor

* probs not gonna work

* undo last commit

* a

* b

* c

* d

* e

* f

* g

* long shot

* 1

* no 7

* da hell

* large bruh moment

* lol

* zzzz

* SSH into CI

* yeet

* Tauri mouse position without new Wry

* go for gold

* Correctly lock `ort` version

* minor fixes

* debounce hover events

* WTF Tauri

* Replace DND hooks with goated versions

* wip frontend stuff

* fix ts

* disable library p2p stuff

* remove Spacedrop dialog + hook up backend

* `useOnDndLeave` working

* Close popover when drag outside

* Allow `openFilePickerDialog` for Spacedrop

* couple of fixes

* empty state

* smh

---------

Co-authored-by: Brendan Allan <brendonovich@outlook.com>
2024-01-08 06:42:17 +00:00
Oscar Beaumont 949707c7f9
Vite upgrades (#1911)
* Upgrade Vite + use SWC React plugin

* Upgrade to type module

* lazy load Sentry

* Lazy load prism

* fix

* Lazy load some of the icons

* fix types

* Fix eslint config

* run lint --fix

* Upgrade Turbo

* Turborepo not happy

* Upgrade Typescript

* Stop complaining Turborepo
2024-01-02 06:26:28 +00:00
Oscar Beaumont 26f610014a
Improve ephemeral walker (#1877)
* `non_indexed::walk` as `impl Stream`

* wip

* this is gooooood

* savepoint

* remove

* Batched stream

* `unsafe_streamed_query`

* nightmare nightmare nightmare nightmare nightmare nightmare

* JS Mutex

* cleanup

* proper error handling

* myCode satisfies Typescript

* Move to npm fork of rspc

* fixes

* rspc more crashy crashy

* Typescript is very disappointed
2024-01-02 06:09:44 +00:00
Julian Braha d18ab1f56e
Add missing timeouts (#1907) 2024-01-01 23:44:48 +00:00
Brendan Allan 5371e9f961
[ENG-1512] Replace sync operation types with single type (#1900)
* replace multiple op types with crdt_operation

* cleanup
2023-12-21 05:58:03 +00:00
Ericson "Fogo" Soares 7c90bcb95b
[ENG-1479] AI Prototype (#1845)
* First draft on image labeling

* Fixing execution providers for other OSs

* Better error handling and shutdown

* Working with shallow media processor

* bruh

* Fix warnings

* Now hooked to media processor job

* Link desktop app with libonnxruntime to avoid TLS error during startup

* Be able to change models on runtime
Revert to use labels table instead of tags

* A bug on a model-less inference

* Show AI labels on Inspector
 - Change yolo inference to use half precision
 - Add labels api to core

* Remove LD_PRELOAD

* Fix race condition on model executor shutdown

* Don't load all images in memory moron

* Embeed yolo model in prod build
 - Change yolo model path to new one relative to executable

* Disable volume watcher on linux, it was crashing the app
 - Invalidate labels when they are updated

* Rust fmt

* Minor changes

* Gate onnxruntime linking to the ai-models feature

* Add build script to sd-server to handle onnxruntime linking workaround

* Move AI stuff to its own crate and normalize deps

* Rust fmt

* Don't regenerate labels unless asked to

* Now blazingly fast

* Bad merge

* Fix

* Fix

* Add backend logic to download extra yolo models

* Add models api route
 - Add api call to get available model version
 - Add api call to change the model version

* Improve new model download logic
 - Add frontend to change image labeler model

* Fix new model downloader

* Fix model select width

* invalidate labels count after media_processor generates a new output

* Rename AI crate and first draft on download notifications

* fix types

---------

Co-authored-by: Vítor Vasconcellos <vasconcellos.dev@gmail.com>
Co-authored-by: Brendan Allan <brendonovich@outlook.com>
2023-12-19 09:28:57 +00:00
Brendan Allan 3cabc9c3a9
Basic actor manager (#1888)
* basic declared actor manager

* put actors in separate file

* clippy

* hopefully clean up some clippy warnings

---------

Co-authored-by: jake <77554505+brxken128@users.noreply.github.com>
2023-12-13 17:49:22 +00:00
Brendan Allan a9da6a0093
[ENG-1495] Pull new instance data when receiving cloud sync operations (#1882)
* add sd-cloud-api

* consume sd-cloud-api & leave instance writing to receiving

* module-ise api crate

* put back logging

* remove 'join cloud library'
2023-12-12 07:47:21 +00:00
Brendan Allan 4fb9028452
Cloud Sync Basics (#1856)
* cloud operation models

* please clippy

* cloud sync table

* cloud ingest works very simply

* move stuff around + stop invalidation from triggering every 10ms

* cleanup

* fix ts

* remove localhost

* re-enable vendored openssl
2023-12-06 16:47:51 +00:00
Oscar Beaumont 89a7f735e5
[ENG-1400] Normalised caching (#1734)
* prototype

* `.normalise` helper + only `String` keys

* implement it for 'search.paths'

* redux devtools

* fix

* refactor backend

* wip: upgrade to rspc fork

* mega cursed

* Upgrade Specta-related stuff

* Upgrade Typescript

* Cache debug page

* bruh

* Fix optimistic library setting

* Cache clearing

* better timeout

* Fix tags page

* bit of cleanup

---------

Co-authored-by: Brendan Allan <brendonovich@outlook.com>
2023-12-05 13:16:03 +00:00
Vítor Vasconcellos 98ff6a7986
[ENG-1471] Fix PDF transparent background (#1844)
Fix PDF transparent background
2023-12-01 00:20:09 +00:00
jake c79c190c78
[ENG-1426] Apply clippy --fix with moderate lints (#1779)
* chore(clippy): fix

* chore: formatting

* fix: add cfg back to macos crate

* remove cfg to see if clippy works

* Restore macOS crate level conditional compilation

* More warns fixing

---------

Co-authored-by: Vítor Vasconcellos <vasconcellos.dev@gmail.com>
Co-authored-by: Ericson Fogo Soares <ericson.ds999@gmail.com>
2023-11-28 17:50:11 +00:00
Vítor Vasconcellos 2f01b21b3d
Fix pdf thumbs having incorrect colors (#1826)
Fix BGR being interpreted as RGB when generation pdf thumbs
 - Disabled some pdf render configs for better performance
2023-11-25 05:09:40 +00:00
jake 59ef3b94de
[ENG-1095] HEIF* and PDF thumbnail orientation (#1815)
* fix horizontal pdf rendering

* don't rotate certain formats (heif*)

* return `u32` from scaling function

* add `avifs` as a valid extension

* add avifs as a valid MIME type

* re-categorize `avifs` as a video (sequence)

* Fix mime-types

* Add Hif extension
 - Cannon uses a special extension for heif in its cameras: https://github.com/digipres/digipres.github.io/blob/master/_sources/registries/fdd/fddXML/fdd000525.xml#L213-L221

* hif is actually a heic

---------

Co-authored-by: Vítor Vasconcellos <vasconcellos.dev@gmail.com>
2023-11-24 22:07:25 +00:00
ameer2468 a7fed83556
[ENG-1413] Full disk access perms (#1791)
* fda wip

* clippy

* add tauri invoke fns for FDA

* fda wip

* clippy

* add tauri invoke fns for FDA

* wip

* fda wip

* clippy

* add tauri invoke fns for FDA

* wip

* wip

* wip fda

* remove imports

* hopefully improve FDA

* execute only on macos

* ts

* ts

* Update Platform.tsx

* Update AddLocationButton.tsx

* remove console log

* fix: fda and add unit tests

* temp commit for Jake

* add fda state and keybind handling (so the frontend is kept up to date)

* update FDA

* update imports

* testing purposes

* Jakes work

* fix fda checks

* work in progress (but not working)

* remove dead files

* attempt #2

* !!!temporarily enable devtools in prod

* remove alert

* show FDA screen but don't require it

* add an FDA button to general client settings

* Update AddLocationButton.tsx

* remove dead code

* unused dep

* old errors

* remove import

* dead code

* dead code + typesafety

* eslint

* remove fda dialog references

* remove mp4 vid

* hopefully fix onboarding for non-macos OSes

* shorter nav

---------

Co-authored-by: jake <77554505+brxken128@users.noreply.github.com>
2023-11-23 20:54:45 +00:00
Vítor Vasconcellos 797e4821b6
Fix some possible pdf thumb rendering concurrency bugs (#1809) 2023-11-23 19:38:17 +00:00
Vítor Vasconcellos 2e6e00bc6d
[ENG-1184, ENG-1286, ENG-1330] Rework native dependencies (+ deb fixes) (#1685)
* Almost working

* Downgrade libplacebo
 - FFMpeg 6.0 uses some now removed deprecated functions

* Use -Oz for zimg

* Fix CI script to run the new ffmpeg build script

* Fix heif step name + Ignore docker cache while building in CI

* Fix Opencl build on linux

* Fix adding incorrect -target argument to linker
 - Update zig for windows target

* Disable opengl for ffmpeg, it only uses it as an outdev, not for processing
 - Disable opengl and directx for libplacebo, ffmpeg only supports vulkan when using it
 - Add WIN32_LEAN_AND_MEAN to global cflags to optimize windows api usage
 - Fix 99-heif.sh incorrect bsdtar flag

* Remove WIN32_LEAN_AND_MEAN from global CFLAGS as that was breaking OpenCL build
 - Fix Dockerfile step for cleaning up the out dir
 - Improve licensing handling

* x86_64 windows and linux builds are working

* Fix aarch64 build for windows and linux

* Fix symbol visibility in linux builds
 - Fix soxr failing to download due to sourcefourge
 - Only patch zimg on windows targets
 - Tell cmake to hide libheif symbols

* Fix Linux .so rpath
 - Add lzo dependency
 - Publish source for the built libs
 - Add warning for missing nasm in tauri.mjs
 - Remove ffmpeg install from setup.sh
 - Add download logic for our linux ffmpeg bundle in preprep.mjs

* Remove jobs, docker doesn't support this

* Fix typing

* Change ffmpeg references to native deps
 - Rename FFMpeg.framework to Spacedrive.framework
 - Centralize the macOS native deps build with the windows and linux one
 - Change the preprep script to only download our native deps
 - Remove old macOS ffmpeg build scripts

* Compress native deps before creating github artifact
 - The zip implementation for github artifact does not mantain symlinks and permissions
 - Remove conditional protoc, it is now always included

* Don't strip dylibs, it was breaking them
 - Only download macOS Framework for darwin targets
 - Fix preprep script
 - Improve README.md for native-deps
 - Fix not finding native-deps src

* Attempt to fix macOS dylib

* Fix macOS dylibs
 - Replace lld.ld64 with apple's own linker
 - Add stages for building apple's compiler tools to use instead of LLVM ones

* Ensure sourced file exists

* All targets should build now
 - Fix environment sourcing in build.sh
 - Some minor improvements to cc.sh
 - Fix incorrect flag in zlib.sh
 - Improve how -f[...] flags are passed to compiler and linker
 - Add more stack hardening flags

* We now can support macOS 11.0 on arm64

* Improve macOS Framework generation
 - Remove installed unused deps
 - Improve cleanup and organization logic in Dockerfile last step
 - Move libav* .dll.a to .lib to fix missing files in windows target
 - Remove apple tools from /srv folder after installation to prevent their files from being copied by other stage steps
 - Create all the necessary symlinks for the macOS targets while building
 - Remove symlink logic for macOS target from preprep.mjs

* Remove native-deps from spacedrive repo
 - It now resides in https://github.com/spacedriveapp/native-deps
 - Modify preprep script to dowload native-deps from new location
 - Remove Github API code from scripts (not needed anymore)
 - Add flock.mjs to allow running tauri.mjs cleanup as soon as cargo finishes building in linux

* Handle flock not present in system
 - Allow macOS to try using flock

* Fix preprep on macOS

* Add script that patch deb to fix errors and warnings raised by lintian

* Fix ctrl+c/ctrl+v typo

* Remove gstreamer1.0-gtk3 from deb dependencies

* eval is evil

* Handle tauri build release with an explicit target in fix-deb.sh

* Preserve environment variables when re-executing fix-deb with sudo

* Only execute fix-deb.sh when building a deb bundle

* Improvements fix-deb.sh

* Improve setup.sh (Add experiemental alpine support)
2023-11-17 19:20:14 +00:00
jake 8e8e355b76
[ENG-1026] Full Disk Access (#1778)
* basic FDA start

* FDA checks and prompt

* lockfile

* misc clippy

* Some small warnings

---------

Co-authored-by: Ericson Fogo Soares <ericson.ds999@gmail.com>
2023-11-15 19:59:50 +00:00
Carter 36cd5bf365
Fix Clippy Warnings (#1776)
Fix Warnings
2023-11-14 20:48:17 +00:00
Oscar Beaumont 5705f40aae
[ENG-1333] Display active P2P listeners (#1644)
* wip

* share the actual port

* fixes

* wip: remove it

* cargo fmt

* thanks @niikeec
2023-11-13 07:02:03 +00:00
Oscar Beaumont 561eacfb6e
[ENG-1201] P2P Error Handling (#1705)
* squash

* wip

* wip

* wip

* wip

* finish it up
2023-11-13 06:58:01 +00:00
Vítor Vasconcellos 7f145c5ee3
[ENG-1359] Revert "Fix pdf thumb crashing app (#1460)" (#1766)
Revert "[ENG-1220] Fix pdf thumb crashing app (#1460)"

This reverts commit 6fd8bd4ad8.
2023-11-12 22:47:10 +00:00
Julian Braha 8ad468b29f
Replace deprecated ConnectionHandler::Error (#1733)
Replace deprecated error type
2023-11-04 16:10:02 +00:00
Oscar Beaumont 6941912891
[ENG-1399] Upgrade Rust workspace + IPv6 mDNS (#1729)
Upgrade Rust workspace + IPv6 mDNS
2023-11-04 10:20:22 +00:00
Julian Braha bf543f45d0
Remove unused dependencies in src-tauri and p2p (#1726)
* Remove unused dependency: window-shadows

* Remove unused dependencies: rmp-serde and arc-swap

* Update Cargo.lock

* remove all unused dependencies

---------

Co-authored-by: Oscar Beaumont <oscar@otbeaumont.me>
2023-11-04 04:56:53 +00:00
Oscar Beaumont 53e4316143
[ENG-1369] Remove broadcast-stream from sd-p2p (#1704)
* let me create pr

* a whole lotta changes

* split `p2p_manager.rs` into smaller files

* the arcpocalypse is over

* minor generic cleanup

* wip removing 'MetadataManager'

* more wip

* wip: i am changing branch

* discovery2 -> discovery

* make it somewhat compile

* more wip

* wip: reassembling manager stream

* state more goodly

* wip

* more wip

* removing generic from sd_p2p::Manager

* reassemble networked libraries

* wip: hooking back up mDNS

* multi-flume wip

* contain bad code to a single file

* p2p_manager_actor + split handlers into file per operation

* cleanup after restructure

* cleaning up more

* wip: reenable resync

* wip: remote identity in connection payload

* track connected clients (required for `service.rs`)

* a big ass iterator

* working towards finishing `service.rs`

* service shutdown

* hook up listen channel in service

* fix address resolution

* merge nlm stuff into LibrariesService

* finish library to service mapping

* less footguns in p2p - seal `PeerId`

* fix previous pr

* p2p state rspc query

* send node events to the frontend

* minor

* wip

* more worky, less crashy

* make spacedrop work + debug state

* fix mdns expiry

* clippy

* other clippy

* remove broadcast type

* merge conflict issue
2023-10-31 04:27:25 +00:00
Oscar Beaumont 02775921ef
[ENG-1318, ENG-1199, ENG-931, ENG-1046] Ability to disable mDNS discovery (#1620)
* let me create pr

* a whole lotta changes

* split `p2p_manager.rs` into smaller files

* the arcpocalypse is over

* minor generic cleanup

* wip removing 'MetadataManager'

* more wip

* wip: i am changing branch

* discovery2 -> discovery

* make it somewhat compile

* more wip

* wip: reassembling manager stream

* state more goodly

* wip

* more wip

* removing generic from sd_p2p::Manager

* reassemble networked libraries

* wip: hooking back up mDNS

* multi-flume wip

* contain bad code to a single file

* p2p_manager_actor + split handlers into file per operation

* cleanup after restructure

* cleaning up more

* wip: reenable resync

* wip: remote identity in connection payload

* track connected clients (required for `service.rs`)

* a big ass iterator

* working towards finishing `service.rs`

* service shutdown

* hook up listen channel in service

* fix address resolution

* merge nlm stuff into LibrariesService

* finish library to service mapping

* less footguns in p2p - seal `PeerId`

* fix previous pr

* p2p state rspc query

* send node events to the frontend

* minor

* wip

* more worky, less crashy

* make spacedrop work + debug state

* fix mdns expiry

* clippy

* other clippy

* remove old tests

* add tests back

---------

Co-authored-by: Brendan Allan <brendonovich@outlook.com>
2023-10-30 08:24:04 +00:00
Ericson "Fogo" Soares 033e61ac33
[ENG-1320 | ENG-1340] Separate thumbnails FS location by library | Make the thumbnails actor report back its progress (#1656)
* Renaming error type on ffmpeg subcrate

* Version manager overhaul

* Reworked thumbnail actor

* Updating sharding scheme

* New migration system for thumbnails

* Updating search to new actor

* Updating custom_uri to new thumbnail improvements

* Updating library to new thumbnail stuff

* LibraryId type alias

* Updating indexer to new thumbnail actor

* Updating watcher to new thumbnail actor

* Update location manager to use LibraryId type alias

* Updating location metadata to new LibraryId type alias

* New LocationPubId type alias

* Updating ephemeral walker to new thumbnail stuff

* Updating media processor to new thumbnail actor

* New thumbnailer actor state manager

* Introducing the concept of job phases

* Segregating the thumbnailer actor worker fn

* Fixes on job pausing

* Processing batches with progress reporting

* Updated actor

* Updated media processor

* Small tweaks

* Updating non indexed walker

* Changing a UI string
2023-10-26 04:48:29 +00:00
Vítor Vasconcellos f7277d602f
Frontend misc fixes and deps update (#1650)
Dependencies overhaul
 - Update dependencies for all projects (except Mobile-only deps)
 - Remove unused dependencies from all projects (except Mobile-only deps)
 - Fix Storybook failing to import sd/ui style
 - Add Node 21 as not supported due to sass-loader not working on it yet
 - Add work-around for new rook version requiring webpack specific global property
 - Fix landing dev not working due to missing default env value on dev
 - Fix some incorrect uses of phosphor-icons non server side icons on server components on landing
 - Fix some incorrect uses of phosphor-icons server side icon on client components on landing
 - Fix landing fail to build on dev due to always required a Github Token to get the latest release
 - Fix new Next.js version not suporting Response.redirect due to immutable Headers
 - Add Gitlab as social link for teams page
 - Update Vítor's team photo
 - Add Vítor's twitter link
 - Fix some warning due to missing useEffect dependencies
 - Remove test-files dir
 - Fix QuickPreview unblurred buttons in Linux
 - Formatting
2023-10-24 07:51:58 +00:00
Oscar Beaumont 758ee5bbe2
[ENG-1332] Fix regression causing P2P to be disabled by default (#1641)
lol, lmao, rofl
2023-10-20 08:57:19 +00:00
Oscar Beaumont 51d99cdb3a
[ENG-1331] Fix app startup without IPv6 network stack (#1638)
can i haz ipv6
2023-10-20 08:56:42 +00:00
Vítor Vasconcellos b827f9cc66
Enable thumbnail support for webp images (#1634)
* Enable thumbnail support for webp images

* Remove webp optional dep from sd-images crate
2023-10-20 03:33:55 +00:00
Oscar Beaumont f67b7e807a
[ENG-1291] enable/disable P2P & configurable port (#1609)
* it works pog

* UI for when networking is disabled

* Fix Clippy issues

* clippy tings
2023-10-18 03:52:54 +00:00
Oscar Beaumont 1b856848e8
[ENG-1310, ENG-1300] Spacedrop better testing + fix zero-sized files (#1606)
* unit tests save lives

* Support for zero-sized files

* fix: clippy

---------

Co-authored-by: jake <77554505+brxken128@users.noreply.github.com>
2023-10-17 06:12:38 +00:00
Oscar Beaumont a7ad24c641
[ENG-625] Spacedrop improvements (#1447)
* Fist fight with the abstraction

* Prevent opening multiple Spacedrop dialogues

* Infinite renders = infinite power

* Comment be gone

* Spacedrop multiple files

* correlate connection id's

* fixes

* thanks Typescript

* Clippy errors + fix cfg's from main
2023-10-17 04:31:42 +00:00
Ericson "Fogo" Soares bfe5a65949
[ENG-1267] Move thumbnails generation away from media processor (#1585)
* New vscode task to start developing

* Updating db in case of library updates just in case

* Done

* Forgot to remove some debug logs

* Rust fmt

* Saving thumbnails processing state on app shutdown

* bruh
2023-10-17 02:09:36 +00:00
Ericson "Fogo" Soares 20fde557d4
[ENG-1225] Put brakes on thumbnail generation for ephemeral locations (#1523)
* Done

* Small nitpick
2023-10-12 04:02:27 +00:00
jake 078490cdd5
[ENG-1143] Media View sort by date taken (#1390)
* clippy -_-

* migrations

* alter the migration so it just renames the `dimensions` field (no db re-creation required)

* remove confusing semver addition for `libheif-sys`

* remove warning on the migration as it's just a rename

* add sort opts for resolution and date image was truly taken

* major serde ckeanup & add epoch_time and pixel_count

* rename, cleanup et optimise

* clippy

* ignore this mess

* bindings

* add explanation to schema

* comment out dt test

* better comment and WIP time

* cleanup rust

* failed timezone attempt

* remove image resolution as a sort by option

* update schema (and rename the dimensions table instead of dropping it)

* just show raw date

* add comments and update bindings

* fix migration hopefully

* fix broken migration

---------

Co-authored-by: Jamie Pine <32987599+jamiepine@users.noreply.github.com>
Co-authored-by: Jamie Pine <ijamespine@me.com>
Co-authored-by: Vítor Vasconcellos <vasconcellos.dev@gmail.com>
2023-10-11 00:58:59 +00:00
Vítor Vasconcellos 6fd8bd4ad8
[ENG-1220] Fix pdf thumb crashing app (#1460)
Fix pdf thumb crashing app
2023-10-11 00:47:36 +00:00
Brendan Allan 49cc098f32
[ENG-927, ENG-735, ENG-766] Fix Updater & Tauri 1.5 (#1361)
* custom updater with toasts

* new state management + updated router route

* tauri-specific update route

* ref

* update in prod only

* change 'Install' to 'Update'

* fix tsconfig

* desktop tauri

* remove tauri patch

* tauri 1.5

* tauri 1.5

* use tauri script

* native-deps

* Rework preprep and tauri script to better support tauri 1.5

* Update to tauri 1.5.1
 - Update workspace and apps/desktop dependencies
 - Fix mustache import, @types/mustache is not compatible with ES imports
 - Replace arm64 with aarch64 in machineID, they should be treated the same and this simplyfies the code

* Fix tauri updater not building due to missing key
 - Fix dmg background not being found
 - Generate an adhoc key for tauri updater with it is enabled and the user is doing a prod build

* Fix ctrl+c/ctrl+v typo

* Normalie @tanstack/react-query version through workspace
 - Use undici in scripts instead of global fetch
 - Fix typecheck

* Fix linux prod and dev builds
 - Improve error handling in tauri.mjs

* Normalize dev deps in workspace
 - Improve linux shared libs setup

* Fix CI and server docker

* Fix windows
 - Remove superfluous envvar

* Attempt to fix server, mobile, deb and release updater

* Attempt to fix deb and mobile again
 - Fix type on deb dependency
 - Enable release deb for aarch64-unknown-linux-gnu

* Github doesn't have arm runners
 - Fix typo in server Dockerfile

* Publish deb and updater artifacts

* remove version from asset name

* update commands

* log release

* Some logs on updater errors

* show updater errors on frontend

* fix desktop ui caching

---------

Co-authored-by: Vítor Vasconcellos <vasconcellos.dev@gmail.com>
Co-authored-by: Ericson Fogo Soares <ericson.ds999@gmail.com>
2023-10-10 07:30:56 +00:00
jake fb8af0832b
[ENG-1096] Image Conversions (#1364)
* derive `Type` on `*Extension`

* update `image`, `webp` and add `rspc::Error` support to the `sd-images` crate

* rename function for verbosity

* cleanup clippy allows and use scaling fn for svg conversions

* remove dead comment

* specify `generic` image extensions and add fn to enumerate all compatible extensions

* re-exports and inline hot provided method

* another inline on a provided method

* `formatter` -> `handler` and return an explicit error if the extension isn't supported, and add `convert_image()`

* `get_ext()` inlined helper fn

* add mutation for conversion and query for getting all supported extensions

* fix `svg` scaling and generation

* update bindings

* rm old const

* temporary test for all exts

* verbosity and no manual drop

* remove dbg test

* big enum and remove `cur` extension as `image` doesn't support it

* add huge extension enum

* hopefully get path from file path id and location id

* main merge fix

* chore: update toml and lockfile

* security: update webp

* comment entire crypto crate for build times

* add bincode, huge enum and custom ser/de

* Revert "comment entire crypto crate for build times"

This reverts commit 458addaad6.

* lockfile

* theoretically working ser/de

* add svg scaling

* fix pdf matching bug and use options

* move image scaling fn

* major consts overhaul with good ser/de

* disable heif checking for now due to new trait api

* make the generic handler truly generic

* fix pdf types

* fix pdf scaling so that it's 120dpi A4 paper size by default - height/aspect res are maintained

* heavy cleanup

* note to myself

* bindings & update core to use the scaling fn from the images crate (reducing code dupes)

* move vulnerable `aovec` to be a dev dep so it stays out of release builds (it's only used in tests)

* remoev exif crate from the core as it's handled in a sep. crate

* silence a clippy warning and update lockfile

* lower the maximum file size a slight amount

* bindings and minor dep bumps

* Some improvements

---------

Co-authored-by: Ericson Fogo Soares <ericson.ds999@gmail.com>
2023-10-10 02:57:37 +00:00
Jamie Pine ed751a46d0
[ENG-944] Sidebar UX Improvements (#1270)
* - added local section to sidebar
- added spacedrop screen, showing local peers
- added placeholder network screen
-removed unused swift package
- created a watcher for system volumes to invalidate ui when drives are added/removed

* clouds

* fix more imports

* see more

* open location if volume is location

* gen assets

* remove log

* [ENG-939, ENG-1173] PDF Thumbnails (#1242)

* sd-pdf

* Process PDF blocking render inside a spawn_blocking
 - Load a single global Pdfium instance

* Migrate pdf thumb logic to sd-images
 - Replace block_in_place with spawn_blocking
 - Only load LibHeif once
 - Allow thumbnailer (both indexed and non-indexed locations) to process documents
 - Disable loading pdf viewer in Inspection in favour of loading it's thumbnail

* Try to load pdfium lib from absolute path

* Revert removed import due to rebase

* Small nitpick and some warnings

---------

Co-authored-by: Ericson Fogo Soares <ericson.ds999@gmail.com>

* [ENG-888] Media view should show current folder downward (#1437)

* Done but ugly

* layout

* Now with a select

---------

Co-authored-by: ameer2468 <33054370+ameer2468@users.noreply.github.com>

* add cool folder thing to inspector + stuff

* fix text color

* fix lock

* fix typescript

* fix ts

---------

Co-authored-by: Utku Bakir <74243531+utkubakir@users.noreply.github.com>
Co-authored-by: Vítor Vasconcellos <vasconcellos.dev@gmail.com>
Co-authored-by: Ericson Fogo Soares <ericson.ds999@gmail.com>
Co-authored-by: ameer2468 <33054370+ameer2468@users.noreply.github.com>
Co-authored-by: Brendan Allan <brendonovich@outlook.com>
2023-10-09 10:11:23 +00:00
Vítor Vasconcellos 556cf1df63
[ENG-939, ENG-1173] PDF Thumbnails (#1242)
* sd-pdf

* Process PDF blocking render inside a spawn_blocking
 - Load a single global Pdfium instance

* Migrate pdf thumb logic to sd-images
 - Replace block_in_place with spawn_blocking
 - Only load LibHeif once
 - Allow thumbnailer (both indexed and non-indexed locations) to process documents
 - Disable loading pdf viewer in Inspection in favour of loading it's thumbnail

* Try to load pdfium lib from absolute path

* Revert removed import due to rebase

* Small nitpick and some warnings

---------

Co-authored-by: Ericson Fogo Soares <ericson.ds999@gmail.com>
2023-10-06 20:41:22 +00:00
Vítor Vasconcellos d75a6e9b12
[ENG-1054] libheif for Linux (plus fixes) + Webp update (#1405)
* Move postinstall script to a preprep script
 - Fix libheif crate failing to build with our libheif
 - Rework CI due to postinstall to preprep changes

* Linux heif build script + Update webp

* Fix ctrl+c/ctrl+v bug

* Improve libheif linux script
 - Add support for linux aarch64
 - Add CI workflow to build libheif for linux
 - Some other misc fixes

* Fix libheif CI requires sudo

* Fix wrong path for libheif build.rs override in Windows

* Fix wrong path manipulations in libheif build script

* 🤦

* Use ubuntu-latest in libheif action
 - Specify glib version in target triple to support old distros
 - Fix libheif artifact publishing

* Fix artifact upload path again

* Add musl support for libheif
 - Remove unused files from libheif artifact
 - Add setup logic for libheif in postinstall script

* Build libheif for linux as a shared lib

* Fix meson not building the correct arch
 - Add logic to get git branch from githubs CI env vars

* libheif finnaly works on linux
 - Make spacedrive binary rpath point to where appimage and deb expects our libs to be
 - Add some logic to tauri.js to convince tauri to bundle our shared libs
 - Work-around appimage bundling step breaking sometimes
 - Add logic to handle sigint in tauri.js to ensure we clean up after ourselves
 - Rename postinstall.mjs to setup.mjs
 - Add logic to setup.mjs to point our dev build to our shared libs in linux

* Fix windows dekstop dev
 - Rename setup.mjs to preprep.mjs

* test cache-factory

* Fix preprep script not parsing the cross compilation target triple and always using the host info to download dependencies
 - Fix action env vars not being correctly passed
 - Remove possibility to pass multiple targests to rust action

* Don't compile mobile crates on desktop targets

* Remove cache-factory pull_request trigger

* remove patched tauri cli

* Use git plumbing command to get remote branch name
 - Fallback to reading .git/HEAD if remote name was not retrieved

* fix type

---------

Co-authored-by: Brendan Allan <brendonovich@outlook.com>
2023-10-03 17:31:33 +00:00
jake 1694ff0f3b
[ENG-1165] Add a security policy and notices to the appropriate READMEs (#1409)
* add security policy

* add a link to it in the main and crypto readmes
2023-09-30 08:31:38 +00:00
Ericson "Fogo" Soares 54c1223256
[ENG-1081] Ignore the case of extensions when matching (#1394)
Done
2023-09-28 17:04:07 +00:00
jake d7c4b1ff1d
Feature/target gate HEIF on Linux (#1312)
* feature/target gate the heif stuff

* more gating

* make clippy happy

* `--cfg linux_heif` override and dep bump

* disable `libheif` on windows temporarily too
2023-09-25 11:18:45 +00:00
Oscar Beaumont ae4c3d0e6b
Fix Android bridge + drop IOS and Android projects (#1336)
* Goodbye Android Project, you won't be missed

* fix ignore ignoring Kotlin code

* Backport IOS `Podfile` + move Rust crates into `modules/sd-core`

* Workaround for issue in OpenSSL build script

* Fix IOS

* Migrate IOS bridge into Expo Native Module

* Optimistic update for library creation

* useFrameworks static

* add key to infoplist

* more keys

* fix `Podfile`

* Remove IOS project from repo

* fix mobile ci

* woops wrong place

* clean package.json

* Make Android's Rust code build again

* Add back in `if_watch` patch for IOS

---------

Co-authored-by: Utku Bakir <74243531+utkubakir@users.noreply.github.com>
2023-09-13 10:07:26 +00:00
Oscar Beaumont 94d8f61de9
Update Rust dependencies (#1314)
* Update most of Rust dependencies

* downgrade notify

* a

* fix Linux?

* Android is painful

* update build.gradle

* The lord Clippy has spoken

* Bruh

---------

Co-authored-by: jake <77554505+brxken128@users.noreply.github.com>
Co-authored-by: Utku Bakir <74243531+utkubakir@users.noreply.github.com>
Co-authored-by: Ericson "Fogo" Soares <ericson.ds999@gmail.com>
2023-09-12 14:03:25 +00:00
jake f421b6883f
Fix builds and make String -> PlusCode validation more generic (Google support) (#1326)
fix builds and make string -> pc matching more generic, so it works with google's format too
2023-09-11 11:35:09 +00:00
jake e2a0878e7a
[ENG-1068] Plus codes / Open Location Code support (#1324)
* bring in rand deps

* SUPER crude pluscode attempt

* working pluscodes!

* some cleanup

* `encodeURIComponent` for URL escaping

---------

Co-authored-by: Oscar Beaumont <oscar@otbeaumont.me>
2023-09-11 04:34:24 +00:00
jake d435141496
[ENG-1084] Correctly format Orientation EXIF data in the UI (#1313)
* impl `Display` and a custom serde serializer

* remove the + as rust already shortens coords to 8SF

* remove manual serde impl and string conversion

* add `formatOrientation` to the `MediaData` part of the inspector

* make requested changes

---------

Co-authored-by: Jamie Pine <32987599+jamiepine@users.noreply.github.com>
2023-09-08 12:21:32 +00:00
jake 7edcbb15d6
[ENG-621][ENG-1074] Trait-based image conversion overhaul (#1307)
* sd-images crate which will support raw/dng, bmp, etc

* more work on the image formatter

* re-work `sd-images`, add svg support, r/g/b and r/g/b/a HEIF image support (will all be async again soon)

* remove `ImageFormatter`, add note about r/g/b/(a) heif impl

* implement the image formatter

* rename the conversion trait and minor cleanups

* isolate heif feature and major cleanup

* very untested raw support

* change fn name to `from_path` (a lot more idiomatic)

* clean up orientation fixing

* heif is no longer forbidden (linux has good heif)

also all extensions are correctly matched in lowercase

* fix builds, ext matching, feature gating

* attempt to fix svg handling?

* raw attempt, quite a few errors

* add comment

* new (untested) attempt

* remove `raw` stuff for now

* replace `sd-svg` with a `ToImage` `SvgHandler` impl

* add some simple math to appropriately scale thumbnails (and bmp/ico support)

* add comments regarding how the math works for image thumbs

* rename the trait to `ImageHandler`
2023-09-07 14:08:17 +00:00
Oscar Beaumont 5a0d0396d0
[ENG-566] Stream files through P2P (#1282)
* Add range to `SpaceblockRequest`

* Determine if file should be served from P2P

* Feature flag serveFromP2P + fix `ServeFrom` logic

* wip: yeet files over P2P

* Just work, argh

* restore backend feature flags after restart

* cleanup
2023-09-06 07:22:04 +00:00
jake a92d6c2faf
[ENG-1039] Apply hard clippy lints to the entire ffmpeg crate (#1281)
* apply clippy lints to the entire ffmpeg crate in hopes of making it safer

* wording

* revert the prisma changes

* fix typo

Co-authored-by: Oscar Beaumont <oscar@otbeaumont.me>

* fix another typo

Co-authored-by: Oscar Beaumont <oscar@otbeaumont.me>

* remove so many `#[must_use]`s

* fix bad merge and hopefully clippy

* clippy please work i beg

* make HEIF_EXTENSIONS always available

---------

Co-authored-by: Oscar Beaumont <oscar@otbeaumont.me>
Co-authored-by: Brendan Allan <brendonovich@outlook.com>
2023-09-05 07:35:26 +00:00
jake 57ac28526e
[ENG-560] P2P Credits (#1297)
add a README with appropriate p2p credits
2023-09-05 07:23:07 +00:00
Ericson "Fogo" Soares 3b8541ef54
[ENG-1028] Media data for ephemeral locations (#1287)
Introducing new getEphemeralMediaData
Fixing some minor stuff
Running pnpm format

Co-authored-by: jake <77554505+brxken128@users.noreply.github.com>
2023-09-04 08:28:45 +00:00
Vítor Vasconcellos b0170b9dba
[ENG-904] Thumbnail for svg (#1220)
* Initial implementation for svg rendering

* Remove unused errors
 - Round up size before allocating Pixmap
 - Re-order some operations

* Finish integrating with Thumbnailer
 - Fix svg thumbnail size
 - Fix incorrect color space while converting tiny_skia::Pixmap to image

* Fix Clippy warns

* Feedback + sd-heif async

* Update implementation to match recent changes to the Thumbnailer
 - Change sd-heif and sd-svg to receive the file data, instead of opening the file internally
 - Update changes to make Thumbnailer and sd-heif more async

* Minor import improvement

* Add missing cfg to gate use of sd_heif only to plataforms that have it enabled
2023-08-31 23:01:45 +00:00
Ericson "Fogo" Soares 39549ef74c
[ENG-953] Zero-byte files occasionally receive a random thumbnail (#1280)
* Better handling empty files

* windows bruh
2023-08-31 01:42:54 +00:00
Oscar Beaumont e591516aff
[ENG-401] Spacedrop betterer (#1273)
* Spacedrop progress on `useP2PEvents`

* Spacedrop Tauri file picker for saving

* Prompt for save location when closing dialog

* quick maths

* Spacedrop progress

* Spacedrop cancellation

* Move accept/cancel dialog to toast

* Toast when rejecting transfer

* yeet

* `useEffect` more like `useBug`
2023-08-30 08:07:23 +00:00
Oscar Beaumont d758977d82
Improve Sync + P2P Integration (#1265)
* Big bruh moment

* whoops

* Less stackoverflowy debug

* stuff

* Fix flawed P2P mDNS instance advertisements

* do sync when connecting with peer

* Sync after pairing

* resync_part2 all the time

* Invalidate all the things

* Invalidate whole React Query on sync event

* emit_messages_flag

* emit_messages_flag

* Backend feature flags + "emitSyncEvents" feature

* Patch `confirm` type cause Tauri cringe

* clippy

* idk but plz work

* bruh

* Fix ComLink bug

* remove log

---------

Co-authored-by: Brendan Allan <brendonovich@outlook.com>
2023-08-29 17:54:58 +00:00
jake c674b7107a
[ENG-288, ENG-601] Media data (image) extraction, thumbnail orientation fix (#1099)
* basic layout

* lock

* add codec to image

* very messy wip

* rm that

* lock and toml

* working perfect exif extraction

* formatting

* migration and formatting

* mostly working

* fix

* set date created properly

* fix tsc

* working media data creation

* fix bad main merge? sorry brendan

* schema, migrations, bindings

* working exif data extraction

* why won't it work

* update migrations

* fix bad merge

* final cleanup

* cleanup migrations

* remove test (that was purely used for testing my code)

* working media data pulling, correct thumbnail orientation

* slightly optimise thumbnail rotation

* rename location to prevent specta clashes

* further improvements (location parsing is still broken)

* fix coordinate parsing i think

* rspc add some todos regarding final cleanup

* further thoughts

* major upgrades

* Some improved handling of errors and async calls

* accordion component

* heavily wip flash refactor

* fix builds all because of a (terrible) merge/move

* annoying missing newline

* i really hate exif

* remove dead code

* further flash progress :D

* docs(media-data): clarification

* minor cleanup

* cleanup and some async removal

* fix location parsing

* remove async (will do proper impl for async eventually) and major cleanup

* big W

* clippy and `FlashMode::Unknown` instead of `Invalid`

* add `NIKON` gps ref support

* comments about gps refs

* commit the submodule -_-

* major cleanup & composite image support

* remove old test image

* major cleanup and easier to use API

* remove old consts

* move `ExifReader` to dedicated module

* Media Data Extractor job and shallow job

* Extracting media data on watcher

* report no exif data on file gracefully

* cleanup errors and doctests

* Merging thumbnailer and media data extractor

* Job name and some strings in UI

* remove reliance on `sd-prisma` within the media data crate

* rename query to be more specific

* custom serializer for `MediaTime`

* tweak to format utc -> naive correctly

* generate migrations

* comment out duration in mobile

* delete test-assets folder

* all optional fields

* fix migrations again

* make requested name changes

* make further requested changes

* remove erroneous files from another wip branch

* updates procedures

* use strings where appropriate

* regen pnpm-lock

* add base layouts for video and audio media data

* use appropriate data types in schema and add audio media data framework

* make requested changes

* general cleanup and renaming of enum

* cleanup media data api

* rename media metadata type

* finishing touches

---------

Co-authored-by: Ericson Soares <ericson.ds999@gmail.com>
Co-authored-by: ameer2468 <33054370+ameer2468@users.noreply.github.com>
Co-authored-by: Jamie Pine <32987599+jamiepine@users.noreply.github.com>
Co-authored-by: Oscar Beaumont <oscar@otbeaumont.me>
Co-authored-by: Utku <74243531+utkubakir@users.noreply.github.com>
Co-authored-by: Brendan Allan <brendonovich@outlook.com>
2023-08-29 17:02:55 +00:00
Aditya 08ba4f917a
Extending QuickPreview functionality with additional filetype support (#1231)
* added some files `standard` mime type

* Used `TEXTViewer` Component to show Code Preview

* Update Thumb.tsx

* added `prismjs`

* removed unnecessary comment

* `CODEViewer` Component for Syntax Highlighting

* formatting

* using **Atom** Theme for `Prism`

* merge text/code viewers & bg-app-focus for prism

currently calling onError and onLoad without an Event argument
that should change but i'm not really sure what to do there

* removed unused imports

* Update index.ts

* `TEXTViewer` to `TextViewer_`

* `TextViewer_` to `TextViewer`

* Don't highlight normal TextFiles

* clean code

* `TEXTViewer` to `TextViewer`

* using tailwind classes more

* doing things correctly.

* installed `prismjs` in interface

* using own scroller

* Update Thumb.tsx

* Add an AbortController to the fetch request
 - Fix onError and onLoad calls
 - Format code

* Fix onError being called when request was aborted due to re-render
 - Fix Compoenent re-rendering loop due to circular reference in useEffect
 - Remove unused imports

* Improve text file serving and code syntax highlight
 - Implement way to identify text files in file-ext crate
 - Do not depend only on the file extension to identify text files in custom_uri
 - Import more prismjs language rules files
 - Add line numbers to TextViewer when rendering code

* Clippy and prettier

* Fix reading zero byte data to Vec
 - Improve empty file handling

* Expand code highlight to more file types
 - Fix 10MB when it should be 10KB
 - Add supported for more code and config files extensions to sd-file-ext
 - Add comlink and vite-plugin-comlink for easy js worker integration
 - Move Prismjs logic to a Worker, because larger files (1000+ lines) where causing the UI to hang
 - Replace line-number prismjs plugin with our own implementation

* Fix uppercase extension name

---------

Co-authored-by: Utku <74243531+utkubakir@users.noreply.github.com>
Co-authored-by: pr <pineapplerind.info@gmail.com>
Co-authored-by: Vítor Vasconcellos <vasconcellos.dev@gmail.com>
2023-08-29 10:47:04 +00:00
Oscar Beaumont 7bf0c0ae4a
[ENG-994] Minor P2P fixes (#1238)
* mdns advertisement on detection

* fix bug in P2P discovery

* Add `P2PEvent::ExpirePeer`

* React properly

* Consistent tracing versions

* better logger config

* Reduce excessive logging from P2P

* Fix panic log hook

* Remove `dbg`

* Fix bug in thumbnailer remover

* wip: Connected peers in UI

* `Sync` is large pain

* init services after library load

* improve error handling of logger

* Sync protocol shutdown

* Add "updater" feature

* fix commands bindings

* nlm state debug query

* nlm debug status

* Make TS happy

* Add `RemoteIdentity` to libraries debug info

* Improve debug data presentation

* Among us level sus

* minor fix panic hook

* Fix EOF issue

* fix instance not being added on pairing

---------

Co-authored-by: Utku <74243531+utkubakir@users.noreply.github.com>
Co-authored-by: Ericson "Fogo" Soares <ericson.ds999@gmail.com>
2023-08-27 06:41:26 +00:00
Vítor Vasconcellos b199c03c1e
Always enable tauri update + Misc Clippy warns fix (#1250)
Always enable tauri update
 - Fix some misc clippy warnings
2023-08-25 22:08:34 +00:00
Jamie Pine b8a9d91d8a
[ENG-1001] Config Kind (#1248)
* config

* cleanup

* cleanup

* clsx
2023-08-25 16:27:54 +00:00
Vítor Vasconcellos 7a7d01307e
Formatting (Prettier + Rustfmt 1.72) (#1246)
Formatting
2023-08-24 20:34:52 +00:00
Ericson "Fogo" Soares 28d106a2d5
[ENG-862, ENG-921] Ephemeral locations (#1092)
* Some initial drafts

* Finising the first draft on non-indexed locations

* Minor tweaks

* Fix warnings

* Adding date_created and date_modified to non indexed path entries

* Add id and path properties to NonIndexedPathItem

* Working ephemeral location (hardcoded home for now)

* Fix UI for ephemeral locations

* Fix windows

* Passing ephemeral thumbnails to thumbnails remover

* Indexing rules for ephemeral paths walking

* Animate Location button when path text overflow it's size

* Fix Linux not showing all volumes

* Fix Linux
 - Add some missing no_os_protected rules for macOS
 - Improve ephemeral location names

* Remove unecessary import

* Fix Mobile

* Improve resizing behaviour for ephemeral location topbar path button
 - Improve Search View (Replace custom empty component with Explorer's emptyNotice )
 - Improve how TopBar children positioning

* Hide EphemeralSection if there is no volume or home
 - Disable Ephemeral topbar path button animation when text is not overflowing

* minor fixes

* Introducing ordering for ephemeral paths

* TS Format

* Ephemeral locations UI fixes
 - Fix indexed Locations having no metadata
 - Remove date indexed/accessed options for sorting Ephemeral locations
 - Remove empty three dots from SideBar element when no settings is linked

* Add tooltip to add location button in ephemeral locations

* Fix indexed Locations selecting other folder/files in Ephemeral location

* Minor fixes

* Fix app breaking due to wrong logic to get item full path in Explorer

* Revert some recent changes to Thumb.tsx

* Fix original not loading for overview items
 - Fix QuickPreview name broken for overview items

* Improve imports

* Revert replace useEffect with useLayoutEffect for locked logic in ListView
It was causing the component to full reload when clicking a header to sort per column

* Changes from feedback

* Hide some unused Inspector metadata fields on NonIndexedPaths
 - Merge formatDate functions while retaining original behaviour

* Use tauri api for getting user home

* Change ThumbType to a string enum to allow for string comparisons

* Improve ObjectKind typing

---------

Co-authored-by: Vítor Vasconcellos <vasconcellos.dev@gmail.com>
Co-authored-by: Oscar Beaumont <oscar@otbeaumont.me>
2023-08-23 17:26:07 +00:00
Brendan Allan 98f763134e
lock libheif-sys version (#1183) 2023-08-07 10:12:20 +00:00
Oscar Beaumont 9e919f9909
Library "state" (#1160)
* `Library` -> `LoadedLibrary`

* Move `ThumbnailRemoverActor` onto `NodeServices`
It should either be on `Library` or `NodeServices` none of this in-between stuff.

* small > big files

* prune imports

* isolate cringe

* it helps to test the code

* `Vec` -> `HashMap` - faster lookups

* mpscrr

* minor fixes

* fix language

* `!Sync` :(

* lets goooooo

* removed a comma

* fixes

* fix desktop

* lol, no one uses it

* Clippy
2023-08-05 14:42:21 +00:00
Oscar Beaumont 6c9e08540f
Networked sync (#1133)
* temp remove discriminator from tunnel

* i'm lost :(

* Working `Tunnel` + move sync stuff back into p2p

* sync module

* `NetworkedSyncManager`

* Move sync stuff off `P2PManager`

* Test `SyncMessage`

* Library edit + delete hooks

* stable `identity` column

* fix

* plz work

* p2p is hard

* sync identities post pairing

* Remove `p384`

* a couple improvements

* comment out ReqRes

---------

Co-authored-by: Brendan Allan <brendonovich@outlook.com>
2023-08-02 11:49:50 +00:00
John Smith 58b7a96860
fix: unknown feature proc_macro_span_shrink (#1143)
fix: https://github.com/rust-lang/rust/issues/113152
2023-08-02 06:02:06 +00:00
Vítor Vasconcellos 7e4ed2b547
[ENG-767, ENG-916] Improvements and fixes for Open/Open With (#1082)
* Normalize PATH and XDG envvars on Linux

* Fix normalize_xdg_environment

* Replace custom Desktop file parsers with Glib

* Fix AppImage env influencing external apps
 - Normalize GStream plugin path

* Fix macos pulling linux deps

* Attempt to fix gnome apps failing to launch
 - Fix incorrect logic in `normalize_pathlist`

* Ensure empty envvars are not set

* Revert AppImage ovewritten GTK specific variables
 - `normalize_pathlist` now prefers entries with less priority when dealing with repeated entries, this is not compatible with the default behavior, but it is a more sane approach IMHO

* Remove 32-bit libs from release CI build host

* Remove 32-bit from github runner 2 attempt
 - Remove deprecated vscode config

* Remove libc6-i386

* [ENG-916] Implement `Open With` logic compatible with macOS < 12

* Add some missing gstreamer deps in Linux

* Replace opener with Glib API

* Fix reveal opening file instead of dir
 - Improve Open With logic in Windows
 - Expose functions to test if app is in a flatpak, snap or appimage

---------

Co-authored-by: Brendan Allan <brendonovich@outlook.com>
2023-07-24 17:36:00 +00:00
Brendan Allan 043b607ad4
CRDTOperation receiving (#1122)
* operation receive + compare

* cleanup + deduplication

* operation receive + compare

* cleanup + deduplication

* sync route + operation grouping

* tag assign sync

* proper relation support in sync debug page

* migration

* separate core-sync + utils crates

* separate p2p event loop from manager

* cleanup library handling

* clippy

* feature gate sync messages properly

* make migration not add required field
2023-07-24 15:26:00 +00:00
Vítor Vasconcellos 8d6a060343
[ENG-914] Fix compilation in Linux arm64 (#1104)
Remove unecessary casts in movie_decoder that were breaking compilation in linux arm64

Co-authored-by: Utku <74243531+utkubakir@users.noreply.github.com>
2023-07-17 17:08:09 +00:00
Brendan Allan 0a1d721e58
Use codegen for ModelSyncType exec (#1098)
* use codegen for ModelSyncType exec

* folder client format

* hande create operation properly
2023-07-15 04:22:04 +00:00
Oscar Beaumont cf39f8dbcc
[ENG-759] P2P Cleanup (#1062)
* less stupid name

* yeet

* awaiting futures is kinda important lol

* no-async

* more proto stuff

* cleanup

* move it round

* None of my homies like broadcast

* minor

* do the shuffle

* restore after force push

* reusing `sysinfo::System` as intended

* fix lol

* remove `node_id` from `Volume`

* Remove `node_local_id` from `Library`

* Remove `Job` to `Node` relation

* feature flags be like

* press save 4head

* remove `Location` -> `Node` relation

* `volume.rs` to `volume/mod.rs`

* yes

* add `Instance` model and deprecate `Node` model

* pairing is better now

* Pretty code

* thinking in code

* wip

* What if Observables but in Rust?!

* Observables aren't it + `useP2PEvents` hook

* more around some jsx

* Trade offer: bad code for working pairing?

* Unit test pairing protocol

* fix everything up

* corrections

* Have you got a moment for our lord and saviour Clippy

* tsc --fixMyCode

* Prisma being wacky

* Config files being f'ed up

* broken config after migrating

* Zed being Zed

* Argh

* cliipzy

* rewind

* Fix truncate logic

* wip: instances in peer metadata

* Rethink instance ids

* fix

* whoops

---------

Co-authored-by: Brendan Allan <brendonovich@outlook.com>
2023-07-12 06:23:30 +00:00
Niklas Wojtkowiak c877c03b63
Add functionality to some Overview Catrgories (#1090)
Co-authored-by: Brendan Allan <brendonovich@outlook.com>
2023-07-12 06:01:30 +00:00
Vítor Vasconcellos caaacc8f92
[ENG-866, ENG-535] Thumbnailer fixes (double-free, thumbnails for portrait videos) (#1059)
* Fix double-free during MovieDecoder Drop
 - Fix broken thumbnails for portrait videos
 - Fix some inconsistencies between our MovieDecoder impl and ffmpegthumbnailer impl
 - Make sd-desktop-macos use the correct macOS deployment target

* fmt

---------

Co-authored-by: Jamie Pine <32987599+jamiepine@users.noreply.github.com>
Co-authored-by: jake <77554505+brxken128@users.noreply.github.com>
Co-authored-by: Utku <74243531+utkubakir@users.noreply.github.com>
2023-06-30 11:00:04 +00:00
Ericson "Fogo" Soares dcd071d229
[ENG-850] Wrongly labelled .ts files (as video) (#1052)
* Fixing conflits of mts code files showing as video

* Introducing a reidentify_objects param on fullRescan

* Fix job output metadata in case of completed with errors
2023-06-29 02:58:33 +00:00
jake 3b86c99d74
[ENG-828-FIX-2] Actually generate thumbnails (#1033)
damn binary operations
2023-06-27 10:55:43 +00:00
jake 9e206b3cc9
[ENG-828-FIX] Make trivial changes to the previous ENG-828 merge (#1032)
remove old error and cleanup
2023-06-27 10:05:47 +00:00