Commit graph

722 commits

Author SHA1 Message Date
Utku Bakir 287f7acc95 Merge branch 'main' into eng-1748-spacedrop-refactor-spacedrop-cloud 2024-05-10 11:45:54 -04:00
Ericson "Fogo" Soares 69412accae
[ENG-1634 / ENG-1636] Integrate Media Data Extractor and Thumbnail Actor with New Task System (#2423)
* Moving stuff around

* Media data extraction task

* New thumbnailer task

* Taking more metrics on thumbnailer

* First drafts on media processor job

* Better interruption latency on thumbnailer task

* Also solve the latency on interrupt at media data extraction

* Media processor job

* Shallow media processor

* Better prioritize files identification

* Integrate ffmpeg media data extraction
- Thumbnail quality improvements parameters
2024-05-10 12:51:22 +00:00
Vítor Vasconcellos 0d451d6d90
Improve Thumbnail quality + fixes (#2467)
* Improve Thumbnail quality
 - Increase thumbnail size to 1024
 - Increse webp quality to 60%

* Fix thumbnails reactivity for ephemeral files

* Fix negative BigInt convertion

* Fix overflow in javascript

---------

Co-authored-by: Ericson Soares <ericson.ds999@gmail.com>
2024-05-09 05:48:43 +00:00
Consoli 8e994bedaa
Box heavy data inside enum (#2466)
* Box big data inside enum

* Ignore size difference of Processed
2024-05-09 04:13:35 +00:00
Vítor Vasconcellos e797b02e65
Media metadata extraction & Thumbnailer rework (#2285)
* initial ffprobe commit

* Working slim down version ffprobe

* Auto format ffprobe and deps source

* Remove show_pixel_formats logic
- Fix do_bitexact incorrect check in main after last changes
- Fix some clangd warning

* Remove show_* and print_format options and their respective logic

* Rework ffprobe into simple_ffprobe
- Simplify ffprobe logic into a simple program that gather and print a media file metadata

* Reduce the amount of ffmpeg log messages while generating thumbnails

* Fix completly wrong comments

* mend

* Start modeling ffmpeg extracted metadata on schema
 - Start porting ffprobe code to rust
 - Rename some references to media_data to exif_data

* Finish modeling media info data
 - Add MediaProgram, MediaStream, MediaCodec, MediaVideoProps, MediaAudioProps, MediaSubtitleProps to Schema
 - Fix simple_ffproble to use its custom print_codec, instead of ffmpeg's impl

* Add relation between MediaInfo and FilePath
 - Remove shared properties from MediaInfo and related structs
 - Implement Iterator for FFmpegDict

* Fix and update schema

* Data models and start populating MediaInfo in rust

* Finish populating media info, chapters and program

* Improve FFmpegFormatContext data raw pointer access
 - Implement stream data gathering

* Impl FFmpegCodecContext, retrieve codec information
 - Improve some unsafe pointer uses
 - Impl from FFmpegFormatContext to MediaInfo conversion

* Fix FFmpegDict Drop

* Fix some crago warnings

* Impl retrieval of video props
 - Fix C char* to Rust String convertion

* Impl retrieval of audio and subtitle props
 - Fill props for MediaCodec

* Remove simple_ffprobe now that the Rust impl is done

* Fix schema to match actually retrieved media info
 - Fix import some FFmpeg constants instead of directly using values

* Rework movie_decoder
 - Re-implement create_scale_string and add support anamorphic video
 - Improve C pointer access for FFmpegFormatContext and FFmpegCodecContext
 - Use newer FFmpeg abstractions in movie_decoder

* Fix incorrect props when initializing MovieDecoder

* Remove unecessary lifetimes

* Added more native wrappers for some FFmpeg native objects used in movie_decoder

* Remove FFmpegPacket
 - Some more improvements to movie_decoder

* WIP

* Some small fixes

* More fixes
Rename movie_decoder to frame_decoder
Remove more references to film_strips

* fmt

* Fix duplicate migration for job error changes

* fix rebase

* Solving segfaults, fuck C lang

Co-authored-by: Vítor Vasconcellos <HeavenVolkoff@users.noreply.github.com>

* Update rust to version 1.77
 - Pin rust version with rust-toolchain.toml
 - Change from dtolnay/rust-toolchain to IronCoreLabs/rust-toolchain for rust-toolchain support
 - Remove unused function and imports
 - Replace most CString uses with new c literal string

* More segfault solving and other minor fixes

Co-authored-by: Vítor Vasconcellos <HeavenVolkoff@users.noreply.github.com>

* Fix ffmpeg rotation filter breaking portrait video thumbnails #2150
 - Plus some other misc fixes

* Auto format

* Retrieve video/audio metadata on frontend

* Auto format

* First draft on ffmpeg data save on db

Co-authored-by: Vítor Vasconcellos <HeavenVolkoff@users.noreply.github.com>

* Fix some incorrect changes to prisma schema

* Some fixes for the FFmpegData schema
 - Expand logic to save FFmpegData to db

* A ton of things

Co-authored-by: Vítor Vasconcellos <HeavenVolkoff@users.noreply.github.com>

* Integrating ffmpeg media data in jobs and API

* Rspc can't BigInt

* 🙄

* Add initial ffmpeg metadata entries to Inspector
 - Fix ephemeral metadata api to match the files metadata api call

* Fix Inspector not showing ffmpeg metadata

* Add bitrate, start time and chapters video metadata to Inspector
- Fix backend BigInt conversion incorrectly using i32 instead of u32
- Change FFmpegFormatContext/FFmpegMetaData bit_rate to i64
- Rename byteSize to humanizeSize
- Expand humanizeSize logic to allow handling bits and Binary units
- Move capitalize to @sd/client utils

* Solving some issues

* Fix ffmpeg probe getting incorrect stream id and breaking database unique constraint
 - Fix humanizeSize breaking when receiving floating numbers
 - Fix incorrect equality in StatCard
 - Fix unhandled error in Dialog when trying to remove an unknown dialog

* fmt

* small improvements
 - Remove some unecessary recursion_limit directive
 - Remove unused app_image releated functions
 - Fix metadata query enabled flag

* Add migration for ffmpeg media data

* Fix cypress test

* Requested changes

* Implement feedback
 - Update locale keys for all languages
 - Add pnpm command to update all language keys

* Fix thumb reactivity in non indexed locations

---------

Co-authored-by: Ericson Soares <ericson.ds999@gmail.com>
Co-authored-by: Vítor Vasconcellos <HeavenVolkoff@users.noreply.github.com>
2024-05-09 02:20:28 +00:00
Vítor Vasconcellos 0f405caf24
Fix CI (#2461)
Use native-tls-vendored for reqwest
 - Update prisma-client-rust
 - Fix mock_instance test
 - Clippy + fmt
 - Improve error handling when uninstall i386 libs in setup-system action
2024-05-08 05:04:25 +00:00
Oscar Beaumont a238760c88
[ENG-1690] Upgrade to Tauri 2.0 (#2433)
* Most of it

* Cleanup menu

* fixes

* Fixed linux compilation, still not working though, as the app panic on start

* Add dialog permissions

* fix search keybind

* Fix linux
∙  - Disable linux-ipc-protocol feature, as that is causing panics due to trying to access some WebKit internal structures outside the main thread

* Update to Ubuntu 22.04, new tauri doesn't support ubuntu 20.04
>  - Add note to download page about deb's distro support

* Remove gen/schemas

* Update linux dependencies in setup.sh

* Fix linux deps again
 - Enable rt-tokio-crypto-rust feature in secret-service so clippy stops complaining

* Clippy + auto fmt

* Fix cache-factory
 - Create a devtools feature for desktop app, which is enabled by default on dev builds

* Fix minor error in patchTauri.mjs

* Fix some envvars that have been renamed on tauri v2

* Dont change the secrets

* Add missing linux dependency

---------

Co-authored-by: Vítor Vasconcellos <vasconcellos.dev@gmail.com>
2024-05-07 08:36:50 +00:00
Brendan Allan 408499229b
Sync settings page (#2460)
sync settings page
2024-05-07 07:39:22 +00:00
Utku Bakir f0e81b9e33 Merge branch 'main' into eng-1748-spacedrop-refactor-spacedrop-cloud 2024-05-06 19:49:20 -04:00
Arnab Chakraborty 42b3174431 Formatting 2024-05-03 00:59:30 -04:00
Arnab Chakraborty b7807be619 Add expiry and password types 2024-05-03 00:50:19 -04:00
Arnab Chakraborty 2881340006 Fix file uploading & Finish up Spacedrop Cloud impl 2024-05-02 16:44:36 -04:00
Utku 952bb2d061
Alpha 0.2.14 (#2445)
alpha
2024-05-02 18:16:01 +00:00
Vítor Vasconcellos 5fe9f02299
Fix BLAKE3 (#2441)
* Update BLAKE3

* Auto fmt

* Revert BLAKE3 to 1.5.0 fork
2024-05-02 09:15:05 +00:00
Utku Bakir 9008a4b8ec Merge branch 'main' into eng-1748-spacedrop-refactor-spacedrop-cloud 2024-05-01 12:02:03 -04:00
Vítor Vasconcellos e4b0aedf64
Auto format/lint and revert some changes made to CI (#2429)
Revert some changes made to CI in #2412
 - Auto lint and format
2024-05-01 13:59:16 +00:00
Brendan Allan 334254abe6
AND search params together properly (#2430) 2024-05-01 07:17:44 +00:00
Arnab Chakraborty 913d178170 Mostly working Cloud Uploading 2024-05-01 00:39:22 -04:00
Brendan Allan 25eb77165c
Update sync model docs + add section about actors (#2424)
update sync model docs + add section about actors
2024-04-30 05:24:14 +00:00
Brendan Allan ce669f203c
Basic sync backfill dialog (#2405)
* basic sync backfill dialog

* only call enableSync once

* rename sync enable to backfill

* disallow closing dialog
2024-04-30 04:42:44 +00:00
Oscar Beaumont ce5e285c2f
Remove normalised cache (#2390)
* yeetus

* fix

* a

* fix

* todo

* fix library stuff

---------

Co-authored-by: Jamie Pine <32987599+jamiepine@users.noreply.github.com>
2024-04-30 00:36:39 +00:00
Vítor Vasconcellos 9384bade61
Revert OpenDAL for ephemeral location (#2399)
* Revert "OpenDAL - Ephemeral Locations (#2283)"

This reverts commit 2848782e8e.

* Format

* Fix some diff problems
2024-04-26 00:29:55 +00:00
Arnab Chakraborty e76ff78f3c
Alpha 0.2.13 (#2394)
bump
2024-04-26 00:29:55 +00:00
Jamie Pine ab46cffa11
Reactive file identification (#2396)
* yes

* Explain mysterious if

* Use id alias just for consistency reasons

* yes

* Rust fmt

* fix ts

---------

Co-authored-by: Ericson "Fogo" Soares <ericson.ds999@gmail.com>
Co-authored-by: Utku Bakir <74243531+utkubakir@users.noreply.github.com>
2024-04-25 21:37:25 +00:00
Vítor Vasconcellos 310eb28e63
Fix cargo test & improve pnpm prep native deps download (#2393)
Couple of fixes
 - Increase `pnpm prep` connection timeout to 5min, to better handle downloading native deps under flaky network conditions
 - Fix `cargo test` and cache-factory CI
 - Clippy and fmt
2024-04-25 17:58:50 +00:00
Jamie Pine b1ffbee9b1
Fix thumbnail generation reactivity (#2392)
fix
2024-04-25 16:14:43 +00:00
Ericson "Fogo" Soares 73f521a3b8
[ENG-1629] Write new file identifier with the task system (#2334)
* Introduce deep vs shallow for indexer tasks with different priorities

* Make job wait to dispatch if it's paused

* Extract file metadata task on file identifier job

* Some initial drafts on object processor task

* Object processor task for file identifier

* File Identifier job and shallow
2024-04-25 04:06:11 +00:00
Vítor Vasconcellos 2c777e53f1
Fix core test (#2386)
* Fix core test

* Import CompressedCRDTOperations

---------

Co-authored-by: Ericson "Fogo" Soares <ericson.ds999@gmail.com>
2024-04-24 23:37:38 +00:00
Utku 3bed56d4d9
Alpha 0.2.12 (#2382)
* pnpm

* alpha 0.2.12

* make pnpm version non strict
2024-04-24 14:25:22 +00:00
Oscar Beaumont 0b6bd050a0
Fix main (#2381)
* fix

* fix
2024-04-24 10:09:18 +00:00
Oscar Beaumont ae6c49b0ba
Improved p2p settings (#2379)
improved p2p settings
2024-04-24 08:43:30 +00:00
Brendan Allan 918c2a987d
Batch ingest sync operations (#2378)
batch ingest sync operations
2024-04-24 08:26:50 +00:00
Oscar Beaumont 643bd3a142
Block size (#2377)
Block size + some Clippy
2024-04-24 08:27:31 +00:00
Arnab Chakraborty b4037d6537
Open Trash from the application (#2338)
* Open Trash from the application

* Working Trash Sidebar Button

* Small UI fixes

* Update common.json

* Move openTrash to Tauri Command instead of RSPC

* format and remove type assertion

---------

Co-authored-by: Utku Bakir <74243531+utkubakir@users.noreply.github.com>
2024-04-22 19:46:10 +00:00
Oscar Beaumont 959ccdfd98
Reintroduce P2P Settings (#2365)
* redo backend to be less cringe

* fixed up
2024-04-22 12:43:44 +00:00
Oscar Beaumont ef969f1ada
Remove indexer rules from ephemeral indexer (#2319)
remove indexer rules from ephemeral indexer
2024-04-22 11:47:47 +00:00
Brendan Allan 548fff1e96
Ignore empty object/filepath search filters (#2371) 2024-04-22 10:29:54 +00:00
Oscar Beaumont 52c5c2bfe7
Show errors creating P2P listeners on startup (#2372)
* do it

* fix accuracy

* `useRef` as god intended
2024-04-22 10:28:35 +00:00
Oscar Beaumont 947354f6c0
Remove files over p2p feature (#2364)
* goodbye

* types

* a
2024-04-20 03:21:20 +00:00
Brendan Allan e4dd2b73ec
[ENG-1709] Smarter params merging in search api (#2346)
do smarter params merging in search api
2024-04-18 14:16:18 +00:00
Brendan Allan 663d1e9f1e
Group create and initial update operations into Create (#2329)
* ignore update after delete in sync

* Group create and initial update operations into Create

* fix ts

* fix sync generator

* fix sync debug route

* oops
2024-04-18 03:53:16 +00:00
Brendan Allan a315dd632d
ignore update after delete in sync (#2327) 2024-04-17 04:57:45 +00:00
Brendan Allan b99a1adfca
[ENG-1740] Add sempahore for generating sync operation timestamps (#2335)
add sempahore for generating sync operation timestamps
2024-04-16 08:14:08 +00:00
Arnab Chakraborty 20b8a2b93b
Disable trash crate on mobile (#2332) 2024-04-15 15:30:33 +00:00
Oscar Beaumont e77a475e61
Date header is not working (#2321)
* wip

* more

* bindings

* show date taken

* improve date lookup and include placeholder

* date taken

---------

Co-authored-by: nikec <nikec.job@gmail.com>
2024-04-15 12:53:11 +00:00
Oscar Beaumont 8be59659d8
DB Vacuum Button (#2312)
* Do the thing

* Retry

* fix types

---------

Co-authored-by: Brendan Allan <brendonovich@outlook.com>
2024-04-12 18:15:56 +00:00
Brendan Allan effee2140d
Tag preferences (#2313)
* tag preferences

* types
2024-04-12 10:17:46 +00:00
Oscar Beaumont 2848782e8e
OpenDAL - Ephemeral Locations (#2283)
* `sd-indexer` crate

* more wip

* wip

* fix

* IndexerRules + error handling

* wip

* Fix it

* Thumbnailer

* Link locations

* Workaround rspc's requiring `Sync` for streams.

* I hate this

* Sorting

* fix

* fix

* fix paths

* Fix double file extension

* Fix thumbnailer

* Clippy

* Fix indexer rules

* fix

* lockfile
2024-04-12 06:51:51 +00:00
Arnab Chakraborty 66063d22c2
[ENG-1365] Move to Trash (#2318)
* Working System Trash Connection

* small changes and prettier

---------

Co-authored-by: Utku Bakir <74243531+utkubakir@users.noreply.github.com>
2024-04-12 05:01:37 +00:00
Brendan Allan 40fa3380e5
[ENG-1722] Numeric sync model IDs (#2298)
* numeric sync model ids

* migration

* fix test compilation
2024-04-11 03:46:30 +00:00