Commit graph

745 commits

Author SHA1 Message Date
Arnab Chakraborty bdd2966071 Merge branch 'main' into mob-106-cloud-sync-for-mobile 2024-06-18 10:41:35 +05:30
Ericson "Fogo" Soares bdc242a852
[ENG-1775] Scan location using new jobs (#2476)
* First draft on task system usage, still missing job system

* Scan location roughly working, a ton of stuff to fix yet

* Updating some deps due to crashes and bugs

* Exposing non critical errors to frontend

* Getting active job reports from job system

* Using boxed opaque type to avoid a downcast issue with generics

* Task system issues discovered on race conditions

* Enable debug

* Fix job report in the job manager

* Fix race condition on steal tasks

* Fixed race condition on task suspend

* Some fixes on job progress reporting and save

* Fixed many race conditions and a hard deadlock
Also some progress report polishing

* Ignore .ts and .mts video files for now

* Some better logs

* bruh

* Internal deadlocks and excess of communication in the task system
- Also better logs

* Bunch of fixes and optimizations

* WIP at fixing file identifier

* Fixed file identifier job
- still need to work on its progress report frontend

* A bunch of polishing

* Fixed some bugs and did more polishing

* Cleanup

* Bridging old and new job systems

* A ton of fixes

* A bunch of bugs related to shutdown and resume

* Indexer and watcher bugs

* Log normalizing

* Fixing CI

* Change error! to warn! on non critical errors log

* Fix redirect to new location

* Type annotation

* Bogus merge resolution on cargo lock
2024-06-17 21:30:57 +00:00
ameer2468 560eea1324 Merge remote-tracking branch 'origin' into mob-106-cloud-sync-for-mobile 2024-06-13 15:04:27 +03:00
Vítor Vasconcellos 405fb023d8
Clean up and update rust dependencies (#2544)
* Clean up and update rust dependencies

* Attempt to fix Windows Error
2024-06-10 19:44:00 +00:00
Arnab Chakraborty efa932de41 New device type passing to auth 2024-06-10 19:26:58 +05:30
Arnab Chakraborty 72f26da11f Formatting 2024-06-10 16:36:30 +05:30
Arnab Chakraborty 449fc43d30 Working Mobile Cloud Sync
Cloud Sync works for Mobile, and the mobile app can sync files from a cloud library, and other clients can access the data from the phone's cloud library.
2024-06-10 14:29:34 +05:30
Vítor Vasconcellos 468bc946ea
Fix nvidia on linux segfault (#2542)
* Workaround nvidia on linux segfault

* Fixes

* Don't vendor dbus

* Make dbus and opener linux exclusive dependencies

* opener is a generic dependency

* Add dbus as a dependency for deb and setup.sh

* Clippy fixes

* Replace gl and glfw with glutin for querying nvidia card

* Replace glutin with wgpu

* Update icrate to fix 'multiple applicable items in scope'
2024-06-07 19:14:50 +00:00
Consoli 76bb317201
Display captured backtrace when available (#2504)
* Display captured backtrace when available

* Helpful log message
2024-06-04 14:22:30 +00:00
Julian Braha ef1be196c1
Cleanup: remove unnecessary #[allow(dead_code)] and #[allow(unused)] attributes (#2535)
Remove unnecessary #[allow(dead_code)] and #[allow(unused)] attributes
2024-06-04 07:11:36 +00:00
Oscar Beaumont 58dd5c5d3e
More More P2P Docs (#2525)
* Docs

* Clarify relay upgrades

* caaaaalapse

* Cleanup `sd_p2p_tunnel`
2024-05-31 07:51:59 +00:00
Brendan Allan 735e80ad4d
[ENG-1776] Delete cloud ops after they've been ingested (#2512)
* delete cloud ops after they've been ingested

* give wait_tx to p2p sync ingest

* remove Ingested event

* add sync docs for setting relation fields

* Update core/crates/sync/README.md

Co-authored-by: Oscar Beaumont <oscar@otbeaumont.me>

---------

Co-authored-by: Oscar Beaumont <oscar@otbeaumont.me>
2024-05-31 06:54:22 +00:00
Utku ab97572878
Alpha 0.3.1 (#2526)
* alpha 0.3.1

* Add missing migration

---------

Co-authored-by: Vítor Vasconcellos <vasconcellos.dev@gmail.com>
2024-05-31 05:44:00 +00:00
Oscar Beaumont 0392c781d7
Serve files over p2p (#2523)
* serve files over p2p

* include location instance id in sync

* Fix P2P addressing?

---------

Co-authored-by: Brendan Allan <brendonovich@outlook.com>
2024-05-30 22:07:28 +08:00
Oscar Beaumont b015763a6f
More P2P docs (#2492)
* Remove relay

* restructure p2p

* wip

* cleanup webrtc

* split up P2P docs

* wip

* more wip

* the fork has moved

* finish local network discovery

* Document the relay system

* be less stupid

* a

* remote ip from deploy script

* remove debug from deploy script

* Explain relay setup and usage

* Physical pain

* fix

* error handling for relay setup

* Listeners Relay state + merge it into NLM state

* `node_remote_identity`

* redo libraries hook

* toggle relay active in settings

* Dedicated network settings page

* Stablise P2P debug page

* warning for rspc remote

* Linear links in docs

* fix p2p settings switches

* fix typescript errors on general page

* fix ipv6 listener status

* discovery method in UI

* Remove p2p debug menu on the sidebar

* wip

* lol

* wat

* fix

* another attempt at fixing library hook

* fix

* Remove sync from sidebar

* fix load library code

* I hate this

* Detect connections over the relay

* fix

* fixes

* a

* fix mDNS

* a bunch o fixes

* a bunch of state management fixes

* Metadata sync on connection

* skill issue

* fix markdown

* Clippy cleanup

* Backport #2380

* Update interface/locales/en/common.json

Co-authored-by: Matthew Yung <117509016+myung03@users.noreply.github.com>

* Update docs/developers/p2p/local-network-discovery.mdx

Co-authored-by: Matthew Yung <117509016+myung03@users.noreply.github.com>

* Update docs/developers/p2p/local-network-discovery.mdx

Co-authored-by: Matthew Yung <117509016+myung03@users.noreply.github.com>

* Update docs/developers/p2p/relay.mdx

Co-authored-by: Matthew Yung <117509016+myung03@users.noreply.github.com>

* Update docs/developers/p2p/relay.mdx

Co-authored-by: Matthew Yung <117509016+myung03@users.noreply.github.com>

* Update docs/developers/p2p/relay.mdx

Co-authored-by: Matthew Yung <117509016+myung03@users.noreply.github.com>

* Update docs/developers/p2p/relay.mdx

Co-authored-by: Matthew Yung <117509016+myung03@users.noreply.github.com>

* Update docs/developers/p2p/relay.mdx

Co-authored-by: Matthew Yung <117509016+myung03@users.noreply.github.com>

* Update docs/developers/p2p/sd_p2p.mdx

Co-authored-by: Matthew Yung <117509016+myung03@users.noreply.github.com>

* Update docs/developers/p2p/sd_p2p.mdx

Co-authored-by: Matthew Yung <117509016+myung03@users.noreply.github.com>

* Update docs/developers/p2p/sd_p2p_proto.mdx

Co-authored-by: Matthew Yung <117509016+myung03@users.noreply.github.com>

* Update docs/developers/p2p/overview.mdx

Co-authored-by: Matthew Yung <117509016+myung03@users.noreply.github.com>

* Update docs/developers/p2p/overview.mdx

Co-authored-by: Matthew Yung <117509016+myung03@users.noreply.github.com>

* Update docs/developers/p2p/relay.mdx

Co-authored-by: Matthew Yung <117509016+myung03@users.noreply.github.com>

* Update docs/developers/p2p/sd_p2p_proto.mdx

Co-authored-by: Matthew Yung <117509016+myung03@users.noreply.github.com>

* Update docs/developers/p2p/sd_p2p_proto.mdx

Co-authored-by: Matthew Yung <117509016+myung03@users.noreply.github.com>

* Update docs/developers/p2p/transport-layer.mdx

Co-authored-by: Matthew Yung <117509016+myung03@users.noreply.github.com>

* Update docs/developers/p2p/sd_p2p_proto.mdx

Co-authored-by: Matthew Yung <117509016+myung03@users.noreply.github.com>

* Update docs/developers/p2p/local-network-discovery.mdx

Co-authored-by: Matthew Yung <117509016+myung03@users.noreply.github.com>

* Update docs/developers/p2p/sd_p2p_proto.mdx

Co-authored-by: Matthew Yung <117509016+myung03@users.noreply.github.com>

* a

* Cleaning binario section

* cleanup Docker message

* idk

* Idempotent listeners

* Manual peers working????

* minor fixes

* crazy idea - don't panic in the event loop

* fixes

* debug

* debug

* LAN badge in network settings

* Use `dns_lookup` instead of `tokio::net::lookup_host`

* fix

* bruh sandwich

* proper dialing

* a

* remove logs

* fix

* Small cleanup

* manual peers state on connected device

* a

* Fix manual discovery state + give it a badge

* Clippy improvements

* flip discovery priority

* Add `addrs` to debug query

* connection candidates in debug

* Fix state

* Clippppppppppppy

* Manual discovery badge

* Flesh out ping example

* Usage guide

* `sd_p2p_proto` examples

* More discovery docs

* More docs work

* docs docs docs and more docs

* PONG

* rename

---------

Co-authored-by: Matthew Yung <117509016+myung03@users.noreply.github.com>
2024-05-30 21:48:12 +08:00
Brendan Allan b7c3eb6f5c
sd-core-sync documentation (#2511)
* sd-core-sync api documentation

* use spaces
2024-05-29 03:30:24 +00:00
Utku a76aca7d3c
alpha 0.3.0 (#2515)
hello world
2024-05-28 21:35:14 +00:00
Arnab Chakraborty a63d66da4c
No Duplicating Tags (#2505)
* Error when trying to make tags with same name

* Formatting
2024-05-24 18:20:20 +00:00
Consoli ce3d4ff7a6
Remove unnecessary allocation (#2495) 2024-05-17 16:10:43 +00:00
ameer2468 9d47af8bd1
[MOB-95] Tags support & more (#2494)
* tags support wip

* Mob: tags in explorer, ui adjustments, add filter based on search click and more

* Fix tags scroll on filters

* Set heights so UI updates correctly

* Update Tags.tsx

* remove console logs

* remove console logs

* Update Locations.tsx

* type

* Update locations.rs

* make tags abit smaller

* list view visual improvements

* Remove plus
2024-05-17 15:55:29 +00:00
Consoli 26b6baffb6
Extend indexer rules with .gitignore when available (#2459)
* Extend indexer rules with `.gitignore` when available

* Make indexer accounts for valid git  repos when reading ignore files

* Ignore poorly formatted gitignore rules

* Reduce allocation in gitignore indexer rules

* Fix tests for gitignore indexer rules

* Remove code duplication

* Remove double call to git rules

* Improve doc for todo

* rustfmt

* Skip star rules that matches a negated pattern when parsing gitignore

* Skip conflicting glob rules

* Rename and remove print

* Make git ignore indexer consider multiple paths

* Use gix_ignore to handle gitignore rules

* Fix gitignore patterns relative to git repo

* Git indexer also consider nested repos

* Create a dedicated type for rules coming from git

* Consider every gitignore file inside a git repo

* Add IgnoredByGit to interface

* Add Gitignore system IndexerRule to allow front-end to toggle GitIgnoreRules
 - Make IndexerRules names unique
 - CLippy fmt

* Fix migration concat

---------

Co-authored-by: Vítor Vasconcellos <vasconcellos.dev@gmail.com>
2024-05-17 06:28:12 +00:00
Utku e7ced995b9
Alpha 0.2.15 (#2491)
alpha 0.2.15
2024-05-16 14:20:32 +00:00
Oscar Beaumont fd10eca115
P2P Docs (#2344)
* Add title to rspc

* wip

* rspc docs title cleanup

* a

* todo

* `p2p/crates`

* remove normalised cache docs

* wip

* words words and more words

* a

* wip

* a

* cleanup settings

* wip p2p docs

* wip
2024-05-14 10:19:33 +00:00
Brendan Allan 7faa08c53a
Make sync ingest setting public (#2485)
make sync ingest setting public
2024-05-14 08:38:00 +00:00
Julian Braha 518d5836f6
p2p: improve error handling for QuicTransport (#2482) 2024-05-14 03:11:25 +00:00
Jamie Pine 8bfcf58a0d
[ENG-1778] Total library size statistic (#2477)
* it

* migration

* fix error msg

* fix error msg actually tho

* fix

* ts errors
2024-05-13 22:02:49 +00:00
Vítor Vasconcellos dcbe888d03
Fix macOS bugs (#2472)
* Fix macOS resize bug
 - Replace custom macOS blur effect with Tauri's V2 native impl

* Fix webview reloading on dev breaking the app

* Fix OpenWith
2024-05-12 06:55:26 +00:00
Ericson "Fogo" Soares 062c71fe41
Fix pnpm prep on heavy-lifting subcrate (#2475)
Fix
2024-05-10 18:25:19 +00:00
Julian Braha 110fef1ce1
Create SpacedropError enum for spacedrop() errors (#2469)
core: spacedrop: use error enum
2024-05-10 15:57:16 +00:00
Brendan Allan d3ce2e8917
use prisma-client-rust main branch (#2471)
* use primsa-client-rust main branch

* remove as i64
2024-05-10 15:54:34 +00:00
Ericson "Fogo" Soares 69412accae
[ENG-1634 / ENG-1636] Integrate Media Data Extractor and Thumbnail Actor with New Task System (#2423)
* Moving stuff around

* Media data extraction task

* New thumbnailer task

* Taking more metrics on thumbnailer

* First drafts on media processor job

* Better interruption latency on thumbnailer task

* Also solve the latency on interrupt at media data extraction

* Media processor job

* Shallow media processor

* Better prioritize files identification

* Integrate ffmpeg media data extraction
- Thumbnail quality improvements parameters
2024-05-10 12:51:22 +00:00
Vítor Vasconcellos 0d451d6d90
Improve Thumbnail quality + fixes (#2467)
* Improve Thumbnail quality
 - Increase thumbnail size to 1024
 - Increse webp quality to 60%

* Fix thumbnails reactivity for ephemeral files

* Fix negative BigInt convertion

* Fix overflow in javascript

---------

Co-authored-by: Ericson Soares <ericson.ds999@gmail.com>
2024-05-09 05:48:43 +00:00
Consoli 8e994bedaa
Box heavy data inside enum (#2466)
* Box big data inside enum

* Ignore size difference of Processed
2024-05-09 04:13:35 +00:00
Vítor Vasconcellos e797b02e65
Media metadata extraction & Thumbnailer rework (#2285)
* initial ffprobe commit

* Working slim down version ffprobe

* Auto format ffprobe and deps source

* Remove show_pixel_formats logic
- Fix do_bitexact incorrect check in main after last changes
- Fix some clangd warning

* Remove show_* and print_format options and their respective logic

* Rework ffprobe into simple_ffprobe
- Simplify ffprobe logic into a simple program that gather and print a media file metadata

* Reduce the amount of ffmpeg log messages while generating thumbnails

* Fix completly wrong comments

* mend

* Start modeling ffmpeg extracted metadata on schema
 - Start porting ffprobe code to rust
 - Rename some references to media_data to exif_data

* Finish modeling media info data
 - Add MediaProgram, MediaStream, MediaCodec, MediaVideoProps, MediaAudioProps, MediaSubtitleProps to Schema
 - Fix simple_ffproble to use its custom print_codec, instead of ffmpeg's impl

* Add relation between MediaInfo and FilePath
 - Remove shared properties from MediaInfo and related structs
 - Implement Iterator for FFmpegDict

* Fix and update schema

* Data models and start populating MediaInfo in rust

* Finish populating media info, chapters and program

* Improve FFmpegFormatContext data raw pointer access
 - Implement stream data gathering

* Impl FFmpegCodecContext, retrieve codec information
 - Improve some unsafe pointer uses
 - Impl from FFmpegFormatContext to MediaInfo conversion

* Fix FFmpegDict Drop

* Fix some crago warnings

* Impl retrieval of video props
 - Fix C char* to Rust String convertion

* Impl retrieval of audio and subtitle props
 - Fill props for MediaCodec

* Remove simple_ffprobe now that the Rust impl is done

* Fix schema to match actually retrieved media info
 - Fix import some FFmpeg constants instead of directly using values

* Rework movie_decoder
 - Re-implement create_scale_string and add support anamorphic video
 - Improve C pointer access for FFmpegFormatContext and FFmpegCodecContext
 - Use newer FFmpeg abstractions in movie_decoder

* Fix incorrect props when initializing MovieDecoder

* Remove unecessary lifetimes

* Added more native wrappers for some FFmpeg native objects used in movie_decoder

* Remove FFmpegPacket
 - Some more improvements to movie_decoder

* WIP

* Some small fixes

* More fixes
Rename movie_decoder to frame_decoder
Remove more references to film_strips

* fmt

* Fix duplicate migration for job error changes

* fix rebase

* Solving segfaults, fuck C lang

Co-authored-by: Vítor Vasconcellos <HeavenVolkoff@users.noreply.github.com>

* Update rust to version 1.77
 - Pin rust version with rust-toolchain.toml
 - Change from dtolnay/rust-toolchain to IronCoreLabs/rust-toolchain for rust-toolchain support
 - Remove unused function and imports
 - Replace most CString uses with new c literal string

* More segfault solving and other minor fixes

Co-authored-by: Vítor Vasconcellos <HeavenVolkoff@users.noreply.github.com>

* Fix ffmpeg rotation filter breaking portrait video thumbnails #2150
 - Plus some other misc fixes

* Auto format

* Retrieve video/audio metadata on frontend

* Auto format

* First draft on ffmpeg data save on db

Co-authored-by: Vítor Vasconcellos <HeavenVolkoff@users.noreply.github.com>

* Fix some incorrect changes to prisma schema

* Some fixes for the FFmpegData schema
 - Expand logic to save FFmpegData to db

* A ton of things

Co-authored-by: Vítor Vasconcellos <HeavenVolkoff@users.noreply.github.com>

* Integrating ffmpeg media data in jobs and API

* Rspc can't BigInt

* 🙄

* Add initial ffmpeg metadata entries to Inspector
 - Fix ephemeral metadata api to match the files metadata api call

* Fix Inspector not showing ffmpeg metadata

* Add bitrate, start time and chapters video metadata to Inspector
- Fix backend BigInt conversion incorrectly using i32 instead of u32
- Change FFmpegFormatContext/FFmpegMetaData bit_rate to i64
- Rename byteSize to humanizeSize
- Expand humanizeSize logic to allow handling bits and Binary units
- Move capitalize to @sd/client utils

* Solving some issues

* Fix ffmpeg probe getting incorrect stream id and breaking database unique constraint
 - Fix humanizeSize breaking when receiving floating numbers
 - Fix incorrect equality in StatCard
 - Fix unhandled error in Dialog when trying to remove an unknown dialog

* fmt

* small improvements
 - Remove some unecessary recursion_limit directive
 - Remove unused app_image releated functions
 - Fix metadata query enabled flag

* Add migration for ffmpeg media data

* Fix cypress test

* Requested changes

* Implement feedback
 - Update locale keys for all languages
 - Add pnpm command to update all language keys

* Fix thumb reactivity in non indexed locations

---------

Co-authored-by: Ericson Soares <ericson.ds999@gmail.com>
Co-authored-by: Vítor Vasconcellos <HeavenVolkoff@users.noreply.github.com>
2024-05-09 02:20:28 +00:00
Vítor Vasconcellos 0f405caf24
Fix CI (#2461)
Use native-tls-vendored for reqwest
 - Update prisma-client-rust
 - Fix mock_instance test
 - Clippy + fmt
 - Improve error handling when uninstall i386 libs in setup-system action
2024-05-08 05:04:25 +00:00
Oscar Beaumont a238760c88
[ENG-1690] Upgrade to Tauri 2.0 (#2433)
* Most of it

* Cleanup menu

* fixes

* Fixed linux compilation, still not working though, as the app panic on start

* Add dialog permissions

* fix search keybind

* Fix linux
∙  - Disable linux-ipc-protocol feature, as that is causing panics due to trying to access some WebKit internal structures outside the main thread

* Update to Ubuntu 22.04, new tauri doesn't support ubuntu 20.04
>  - Add note to download page about deb's distro support

* Remove gen/schemas

* Update linux dependencies in setup.sh

* Fix linux deps again
 - Enable rt-tokio-crypto-rust feature in secret-service so clippy stops complaining

* Clippy + auto fmt

* Fix cache-factory
 - Create a devtools feature for desktop app, which is enabled by default on dev builds

* Fix minor error in patchTauri.mjs

* Fix some envvars that have been renamed on tauri v2

* Dont change the secrets

* Add missing linux dependency

---------

Co-authored-by: Vítor Vasconcellos <vasconcellos.dev@gmail.com>
2024-05-07 08:36:50 +00:00
Brendan Allan 408499229b
Sync settings page (#2460)
sync settings page
2024-05-07 07:39:22 +00:00
Utku 952bb2d061
Alpha 0.2.14 (#2445)
alpha
2024-05-02 18:16:01 +00:00
Vítor Vasconcellos 5fe9f02299
Fix BLAKE3 (#2441)
* Update BLAKE3

* Auto fmt

* Revert BLAKE3 to 1.5.0 fork
2024-05-02 09:15:05 +00:00
Vítor Vasconcellos e4b0aedf64
Auto format/lint and revert some changes made to CI (#2429)
Revert some changes made to CI in #2412
 - Auto lint and format
2024-05-01 13:59:16 +00:00
Brendan Allan 334254abe6
AND search params together properly (#2430) 2024-05-01 07:17:44 +00:00
Brendan Allan 25eb77165c
Update sync model docs + add section about actors (#2424)
update sync model docs + add section about actors
2024-04-30 05:24:14 +00:00
Brendan Allan ce669f203c
Basic sync backfill dialog (#2405)
* basic sync backfill dialog

* only call enableSync once

* rename sync enable to backfill

* disallow closing dialog
2024-04-30 04:42:44 +00:00
Oscar Beaumont ce5e285c2f
Remove normalised cache (#2390)
* yeetus

* fix

* a

* fix

* todo

* fix library stuff

---------

Co-authored-by: Jamie Pine <32987599+jamiepine@users.noreply.github.com>
2024-04-30 00:36:39 +00:00
Vítor Vasconcellos 9384bade61
Revert OpenDAL for ephemeral location (#2399)
* Revert "OpenDAL - Ephemeral Locations (#2283)"

This reverts commit 2848782e8e.

* Format

* Fix some diff problems
2024-04-26 00:29:55 +00:00
Arnab Chakraborty e76ff78f3c
Alpha 0.2.13 (#2394)
bump
2024-04-26 00:29:55 +00:00
Jamie Pine ab46cffa11
Reactive file identification (#2396)
* yes

* Explain mysterious if

* Use id alias just for consistency reasons

* yes

* Rust fmt

* fix ts

---------

Co-authored-by: Ericson "Fogo" Soares <ericson.ds999@gmail.com>
Co-authored-by: Utku Bakir <74243531+utkubakir@users.noreply.github.com>
2024-04-25 21:37:25 +00:00
Vítor Vasconcellos 310eb28e63
Fix cargo test & improve pnpm prep native deps download (#2393)
Couple of fixes
 - Increase `pnpm prep` connection timeout to 5min, to better handle downloading native deps under flaky network conditions
 - Fix `cargo test` and cache-factory CI
 - Clippy and fmt
2024-04-25 17:58:50 +00:00
Jamie Pine b1ffbee9b1
Fix thumbnail generation reactivity (#2392)
fix
2024-04-25 16:14:43 +00:00
Ericson "Fogo" Soares 73f521a3b8
[ENG-1629] Write new file identifier with the task system (#2334)
* Introduce deep vs shallow for indexer tasks with different priorities

* Make job wait to dispatch if it's paused

* Extract file metadata task on file identifier job

* Some initial drafts on object processor task

* Object processor task for file identifier

* File Identifier job and shallow
2024-04-25 04:06:11 +00:00