spacedrive/core/src/lib.rs

208 lines
5.8 KiB
Rust
Raw Normal View History

use api::{CoreEvent, Ctx, Router};
use job::JobManager;
use library::LibraryManager;
ENG 223 Location Awareness (#468) * Introducing location online/offline checks and location relink * Some initial drafts for location watchers * Location metadata relink and add library * Many improvements at job system Now using prisma batching at identifier job Removing blocking I/O from extension subcrate Implementing lazy generation of thumbnails New current directory identifier job to be used on light rescans * Some optimizations on identifier and object validator jobs * merge jamie's identifier PR * fully repaired identifier job * properly hooked up object kind * inspector fix * fix video badge * small improvements to libraries settings * identifier and inspector improvements * fix feature flags and hook up context menu location utilities * BETTER CONTEXT MENU x100 * test-files * style tweaks * new icon designs * manifest * fix thumbnails on web * media data * New Location Watcher and some minor fixes * disable broken media_data extractor, wip * wip * function name fix * Fixing pnpm prep and some warnings * Solving a race condition beetween indexer job and FS event handlerSome other minor warnings * Generating thumbnails on watcher * Remove event handler on watcher * Some initial works on modify events and other small fixes * File update event * Trying to be more generic with used events and some tests to validate our assumptions * Turning on location metadata file * Introducing core unit tests on CI pipeline * Submiting new unit test assumptions to validate on windows CI * Fixing unit tests * Fixing unit tests again * Fixing unit tests * Fixing unit tests for macos * Fixing unit tests for macos again * New structure for platform dependent event handling Implementing event handlers for Linux and MacOS * minor fixes + rustfmt + clippy * Windows event handling * Introducing a feature gate to only use location watching on desktop app for now * Putting more stuff behind feature gates to avoid warnings * Adding feature to cargo test on CI * Changing some debug logs to trace logs and removing Jamie specific stuff * Make location removal from manager less async * fix build when "location-watcher" feature disabled * fix types + clippy * make location manager non-static * remove uses of `to_string_lossy` * more invalidate_query calls * Expose `library_ctx` directly to avoid needless clones * New materialized_path handling for directories * Removing cascade delete between file_path and object - Some other minor stuff * remove unused `CurrentDirFileIdentifierJob` Co-authored-by: Jamie Pine <ijamespine@me.com> Co-authored-by: Oscar Beaumont <oscar@otbeaumont.me>
2022-12-30 16:53:24 +00:00
use location::{LocationManager, LocationManagerError};
use node::NodeConfigManager;
[ENG 239] Onboarding Flow & Location Settings (#529) * begin better onboarding * added input and altered text * better router & text + database icon Co-authored-by: maxichrome <maxichrome@users.noreply.github.com> * work on privacy screen + radio buttons * fix video extension bug and alter screens * add pending schema and location manager helper * functional onboarding * added secure temp store and started creating library loading screen * fix secure temp keystore + api * better onboarding * added location settings and some overview concept, all WIP * fix switch * prep * fix location router * added backend settings * attempted to fix form * begin indexer rules editor, plus tweaks * indexer rules coming soon * fix onboarding img size * cleanup * clone is needed here, but clippy no like * sike * whole bunch of fixes * clippy + ts * Removing some TODOs from api/libraries.rs and fixing db size calculation * moved object kind to client, added half functionality for appearance settings * fix RadioGroup helper * fix type issues * cargo fmt * fix creating library error handling + invalidate location list on update * forgot to switch back to onError * Invalidating getStatistics query on library creation and introducing the concept of waiting for a job on FileCopierJob * F* cargo fmt * fix RadioGroup interactivity * wipe all migrations * put back COLLATE NOCASE on extension columns * update core.ts * remove unused device component * fix typeerror in mobile --------- Co-authored-by: maxichrome <maxichrome@users.noreply.github.com> Co-authored-by: Brendan Allan <brendonovich@outlook.com> Co-authored-by: Ericson Soares <ericson.ds999@gmail.com> Co-authored-by: Utku Bakir <74243531+utkubakir@users.noreply.github.com>
2023-02-10 22:08:13 +00:00
use util::secure_temp_keystore::SecureTempKeystore;
ENG 223 Location Awareness (#468) * Introducing location online/offline checks and location relink * Some initial drafts for location watchers * Location metadata relink and add library * Many improvements at job system Now using prisma batching at identifier job Removing blocking I/O from extension subcrate Implementing lazy generation of thumbnails New current directory identifier job to be used on light rescans * Some optimizations on identifier and object validator jobs * merge jamie's identifier PR * fully repaired identifier job * properly hooked up object kind * inspector fix * fix video badge * small improvements to libraries settings * identifier and inspector improvements * fix feature flags and hook up context menu location utilities * BETTER CONTEXT MENU x100 * test-files * style tweaks * new icon designs * manifest * fix thumbnails on web * media data * New Location Watcher and some minor fixes * disable broken media_data extractor, wip * wip * function name fix * Fixing pnpm prep and some warnings * Solving a race condition beetween indexer job and FS event handlerSome other minor warnings * Generating thumbnails on watcher * Remove event handler on watcher * Some initial works on modify events and other small fixes * File update event * Trying to be more generic with used events and some tests to validate our assumptions * Turning on location metadata file * Introducing core unit tests on CI pipeline * Submiting new unit test assumptions to validate on windows CI * Fixing unit tests * Fixing unit tests again * Fixing unit tests * Fixing unit tests for macos * Fixing unit tests for macos again * New structure for platform dependent event handling Implementing event handlers for Linux and MacOS * minor fixes + rustfmt + clippy * Windows event handling * Introducing a feature gate to only use location watching on desktop app for now * Putting more stuff behind feature gates to avoid warnings * Adding feature to cargo test on CI * Changing some debug logs to trace logs and removing Jamie specific stuff * Make location removal from manager less async * fix build when "location-watcher" feature disabled * fix types + clippy * make location manager non-static * remove uses of `to_string_lossy` * more invalidate_query calls * Expose `library_ctx` directly to avoid needless clones * New materialized_path handling for directories * Removing cascade delete between file_path and object - Some other minor stuff * remove unused `CurrentDirFileIdentifierJob` Co-authored-by: Jamie Pine <ijamespine@me.com> Co-authored-by: Oscar Beaumont <oscar@otbeaumont.me>
2022-12-30 16:53:24 +00:00
use std::{path::Path, sync::Arc};
use thiserror::Error;
use tokio::{fs, sync::broadcast};
use tracing::{debug, error, info};
use tracing_subscriber::{prelude::*, EnvFilter};
pub mod api;
pub mod custom_uri;
pub(crate) mod job;
pub(crate) mod library;
pub(crate) mod location;
pub(crate) mod node;
2022-09-04 20:00:24 +00:00
pub(crate) mod object;
2023-01-05 07:11:55 +00:00
pub(crate) mod sync;
pub(crate) mod util;
pub(crate) mod volume;
pub(crate) mod prisma;
2023-01-05 07:11:55 +00:00
pub(crate) mod prisma_sync;
#[derive(Clone)]
2022-07-11 02:05:24 +00:00
pub struct NodeContext {
pub config: Arc<NodeConfigManager>,
pub jobs: Arc<JobManager>,
ENG 223 Location Awareness (#468) * Introducing location online/offline checks and location relink * Some initial drafts for location watchers * Location metadata relink and add library * Many improvements at job system Now using prisma batching at identifier job Removing blocking I/O from extension subcrate Implementing lazy generation of thumbnails New current directory identifier job to be used on light rescans * Some optimizations on identifier and object validator jobs * merge jamie's identifier PR * fully repaired identifier job * properly hooked up object kind * inspector fix * fix video badge * small improvements to libraries settings * identifier and inspector improvements * fix feature flags and hook up context menu location utilities * BETTER CONTEXT MENU x100 * test-files * style tweaks * new icon designs * manifest * fix thumbnails on web * media data * New Location Watcher and some minor fixes * disable broken media_data extractor, wip * wip * function name fix * Fixing pnpm prep and some warnings * Solving a race condition beetween indexer job and FS event handlerSome other minor warnings * Generating thumbnails on watcher * Remove event handler on watcher * Some initial works on modify events and other small fixes * File update event * Trying to be more generic with used events and some tests to validate our assumptions * Turning on location metadata file * Introducing core unit tests on CI pipeline * Submiting new unit test assumptions to validate on windows CI * Fixing unit tests * Fixing unit tests again * Fixing unit tests * Fixing unit tests for macos * Fixing unit tests for macos again * New structure for platform dependent event handling Implementing event handlers for Linux and MacOS * minor fixes + rustfmt + clippy * Windows event handling * Introducing a feature gate to only use location watching on desktop app for now * Putting more stuff behind feature gates to avoid warnings * Adding feature to cargo test on CI * Changing some debug logs to trace logs and removing Jamie specific stuff * Make location removal from manager less async * fix build when "location-watcher" feature disabled * fix types + clippy * make location manager non-static * remove uses of `to_string_lossy` * more invalidate_query calls * Expose `library_ctx` directly to avoid needless clones * New materialized_path handling for directories * Removing cascade delete between file_path and object - Some other minor stuff * remove unused `CurrentDirFileIdentifierJob` Co-authored-by: Jamie Pine <ijamespine@me.com> Co-authored-by: Oscar Beaumont <oscar@otbeaumont.me>
2022-12-30 16:53:24 +00:00
pub location_manager: Arc<LocationManager>,
pub event_bus_tx: broadcast::Sender<CoreEvent>,
}
2022-05-10 16:03:20 +00:00
pub struct Node {
2022-07-11 02:05:24 +00:00
config: Arc<NodeConfigManager>,
library_manager: Arc<LibraryManager>,
jobs: Arc<JobManager>,
event_bus: (broadcast::Sender<CoreEvent>, broadcast::Receiver<CoreEvent>),
[ENG 239] Onboarding Flow & Location Settings (#529) * begin better onboarding * added input and altered text * better router & text + database icon Co-authored-by: maxichrome <maxichrome@users.noreply.github.com> * work on privacy screen + radio buttons * fix video extension bug and alter screens * add pending schema and location manager helper * functional onboarding * added secure temp store and started creating library loading screen * fix secure temp keystore + api * better onboarding * added location settings and some overview concept, all WIP * fix switch * prep * fix location router * added backend settings * attempted to fix form * begin indexer rules editor, plus tweaks * indexer rules coming soon * fix onboarding img size * cleanup * clone is needed here, but clippy no like * sike * whole bunch of fixes * clippy + ts * Removing some TODOs from api/libraries.rs and fixing db size calculation * moved object kind to client, added half functionality for appearance settings * fix RadioGroup helper * fix type issues * cargo fmt * fix creating library error handling + invalidate location list on update * forgot to switch back to onError * Invalidating getStatistics query on library creation and introducing the concept of waiting for a job on FileCopierJob * F* cargo fmt * fix RadioGroup interactivity * wipe all migrations * put back COLLATE NOCASE on extension columns * update core.ts * remove unused device component * fix typeerror in mobile --------- Co-authored-by: maxichrome <maxichrome@users.noreply.github.com> Co-authored-by: Brendan Allan <brendonovich@outlook.com> Co-authored-by: Ericson Soares <ericson.ds999@gmail.com> Co-authored-by: Utku Bakir <74243531+utkubakir@users.noreply.github.com>
2023-02-10 22:08:13 +00:00
secure_temp_keystore: Arc<SecureTempKeystore>,
}
2022-02-26 01:56:49 +00:00
#[cfg(not(feature = "android"))]
const CONSOLE_LOG_FILTER: tracing_subscriber::filter::LevelFilter = {
use tracing_subscriber::filter::LevelFilter;
match cfg!(debug_assertions) {
true => LevelFilter::DEBUG,
false => LevelFilter::INFO,
}
};
2022-05-10 16:03:20 +00:00
impl Node {
pub async fn new(data_dir: impl AsRef<Path>) -> Result<(Arc<Node>, Arc<Router>), NodeError> {
let data_dir = data_dir.as_ref();
ENG 223 Location Awareness (#468) * Introducing location online/offline checks and location relink * Some initial drafts for location watchers * Location metadata relink and add library * Many improvements at job system Now using prisma batching at identifier job Removing blocking I/O from extension subcrate Implementing lazy generation of thumbnails New current directory identifier job to be used on light rescans * Some optimizations on identifier and object validator jobs * merge jamie's identifier PR * fully repaired identifier job * properly hooked up object kind * inspector fix * fix video badge * small improvements to libraries settings * identifier and inspector improvements * fix feature flags and hook up context menu location utilities * BETTER CONTEXT MENU x100 * test-files * style tweaks * new icon designs * manifest * fix thumbnails on web * media data * New Location Watcher and some minor fixes * disable broken media_data extractor, wip * wip * function name fix * Fixing pnpm prep and some warnings * Solving a race condition beetween indexer job and FS event handlerSome other minor warnings * Generating thumbnails on watcher * Remove event handler on watcher * Some initial works on modify events and other small fixes * File update event * Trying to be more generic with used events and some tests to validate our assumptions * Turning on location metadata file * Introducing core unit tests on CI pipeline * Submiting new unit test assumptions to validate on windows CI * Fixing unit tests * Fixing unit tests again * Fixing unit tests * Fixing unit tests for macos * Fixing unit tests for macos again * New structure for platform dependent event handling Implementing event handlers for Linux and MacOS * minor fixes + rustfmt + clippy * Windows event handling * Introducing a feature gate to only use location watching on desktop app for now * Putting more stuff behind feature gates to avoid warnings * Adding feature to cargo test on CI * Changing some debug logs to trace logs and removing Jamie specific stuff * Make location removal from manager less async * fix build when "location-watcher" feature disabled * fix types + clippy * make location manager non-static * remove uses of `to_string_lossy` * more invalidate_query calls * Expose `library_ctx` directly to avoid needless clones * New materialized_path handling for directories * Removing cascade delete between file_path and object - Some other minor stuff * remove unused `CurrentDirFileIdentifierJob` Co-authored-by: Jamie Pine <ijamespine@me.com> Co-authored-by: Oscar Beaumont <oscar@otbeaumont.me>
2022-12-30 16:53:24 +00:00
// This error is ignored because it's throwing on mobile despite the folder existing.
let _ = fs::create_dir_all(&data_dir).await;
// dbg!(get_object_kind_from_extension("png"));
2022-10-06 21:51:40 +00:00
// let (non_blocking, _guard) = tracing_appender::non_blocking(rolling::daily(
// Path::new(&data_dir).join("logs"),
// "log",
// ));
2022-10-06 21:32:45 +00:00
// TODO: Make logs automatically delete after x time https://github.com/tokio-rs/tracing/pull/2169
2022-10-18 12:56:16 +00:00
let subscriber = tracing_subscriber::registry().with(
EnvFilter::from_default_env()
.add_directive("warn".parse().expect("Error invalid tracing directive!"))
.add_directive(
ENG 223 Location Awareness (#468) * Introducing location online/offline checks and location relink * Some initial drafts for location watchers * Location metadata relink and add library * Many improvements at job system Now using prisma batching at identifier job Removing blocking I/O from extension subcrate Implementing lazy generation of thumbnails New current directory identifier job to be used on light rescans * Some optimizations on identifier and object validator jobs * merge jamie's identifier PR * fully repaired identifier job * properly hooked up object kind * inspector fix * fix video badge * small improvements to libraries settings * identifier and inspector improvements * fix feature flags and hook up context menu location utilities * BETTER CONTEXT MENU x100 * test-files * style tweaks * new icon designs * manifest * fix thumbnails on web * media data * New Location Watcher and some minor fixes * disable broken media_data extractor, wip * wip * function name fix * Fixing pnpm prep and some warnings * Solving a race condition beetween indexer job and FS event handlerSome other minor warnings * Generating thumbnails on watcher * Remove event handler on watcher * Some initial works on modify events and other small fixes * File update event * Trying to be more generic with used events and some tests to validate our assumptions * Turning on location metadata file * Introducing core unit tests on CI pipeline * Submiting new unit test assumptions to validate on windows CI * Fixing unit tests * Fixing unit tests again * Fixing unit tests * Fixing unit tests for macos * Fixing unit tests for macos again * New structure for platform dependent event handling Implementing event handlers for Linux and MacOS * minor fixes + rustfmt + clippy * Windows event handling * Introducing a feature gate to only use location watching on desktop app for now * Putting more stuff behind feature gates to avoid warnings * Adding feature to cargo test on CI * Changing some debug logs to trace logs and removing Jamie specific stuff * Make location removal from manager less async * fix build when "location-watcher" feature disabled * fix types + clippy * make location manager non-static * remove uses of `to_string_lossy` * more invalidate_query calls * Expose `library_ctx` directly to avoid needless clones * New materialized_path handling for directories * Removing cascade delete between file_path and object - Some other minor stuff * remove unused `CurrentDirFileIdentifierJob` Co-authored-by: Jamie Pine <ijamespine@me.com> Co-authored-by: Oscar Beaumont <oscar@otbeaumont.me>
2022-12-30 16:53:24 +00:00
"sd_core=debug"
2022-10-18 12:56:16 +00:00
.parse()
.expect("Error invalid tracing directive!"),
)
.add_directive(
"sd_core::location::manager=info"
.parse()
.expect("Error invalid tracing directive!"),
)
2022-10-18 12:56:16 +00:00
.add_directive(
ENG 223 Location Awareness (#468) * Introducing location online/offline checks and location relink * Some initial drafts for location watchers * Location metadata relink and add library * Many improvements at job system Now using prisma batching at identifier job Removing blocking I/O from extension subcrate Implementing lazy generation of thumbnails New current directory identifier job to be used on light rescans * Some optimizations on identifier and object validator jobs * merge jamie's identifier PR * fully repaired identifier job * properly hooked up object kind * inspector fix * fix video badge * small improvements to libraries settings * identifier and inspector improvements * fix feature flags and hook up context menu location utilities * BETTER CONTEXT MENU x100 * test-files * style tweaks * new icon designs * manifest * fix thumbnails on web * media data * New Location Watcher and some minor fixes * disable broken media_data extractor, wip * wip * function name fix * Fixing pnpm prep and some warnings * Solving a race condition beetween indexer job and FS event handlerSome other minor warnings * Generating thumbnails on watcher * Remove event handler on watcher * Some initial works on modify events and other small fixes * File update event * Trying to be more generic with used events and some tests to validate our assumptions * Turning on location metadata file * Introducing core unit tests on CI pipeline * Submiting new unit test assumptions to validate on windows CI * Fixing unit tests * Fixing unit tests again * Fixing unit tests * Fixing unit tests for macos * Fixing unit tests for macos again * New structure for platform dependent event handling Implementing event handlers for Linux and MacOS * minor fixes + rustfmt + clippy * Windows event handling * Introducing a feature gate to only use location watching on desktop app for now * Putting more stuff behind feature gates to avoid warnings * Adding feature to cargo test on CI * Changing some debug logs to trace logs and removing Jamie specific stuff * Make location removal from manager less async * fix build when "location-watcher" feature disabled * fix types + clippy * make location manager non-static * remove uses of `to_string_lossy` * more invalidate_query calls * Expose `library_ctx` directly to avoid needless clones * New materialized_path handling for directories * Removing cascade delete between file_path and object - Some other minor stuff * remove unused `CurrentDirFileIdentifierJob` Co-authored-by: Jamie Pine <ijamespine@me.com> Co-authored-by: Oscar Beaumont <oscar@otbeaumont.me>
2022-12-30 16:53:24 +00:00
"sd_core_mobile=debug"
2022-10-18 12:56:16 +00:00
.parse()
.expect("Error invalid tracing directive!"),
)
.add_directive(
"server=debug"
.parse()
.expect("Error invalid tracing directive!"),
)
.add_directive(
"desktop=debug"
.parse()
.expect("Error invalid tracing directive!"),
), // .add_directive(
// "rspc=debug"
// .parse()
// .expect("Error invalid tracing directive!"),
// ),
);
#[cfg(not(feature = "android"))]
let subscriber = subscriber.with(tracing_subscriber::fmt::layer().with_filter(CONSOLE_LOG_FILTER));
2022-10-18 12:56:16 +00:00
#[cfg(feature = "android")]
let subscriber = subscriber.with(tracing_android::layer("com.spacedrive.app").unwrap()); // TODO: This is not working
subscriber
2022-10-06 21:32:45 +00:00
// .with(
// Layer::default()
// .with_writer(non_blocking)
// .with_ansi(false)
// .with_filter(LevelFilter::DEBUG),
// )
.init();
2022-05-23 04:07:35 +00:00
let event_bus = broadcast::channel(1024);
let config = NodeConfigManager::new(data_dir.to_path_buf()).await?;
2022-07-11 02:05:24 +00:00
let jobs = JobManager::new();
ENG 223 Location Awareness (#468) * Introducing location online/offline checks and location relink * Some initial drafts for location watchers * Location metadata relink and add library * Many improvements at job system Now using prisma batching at identifier job Removing blocking I/O from extension subcrate Implementing lazy generation of thumbnails New current directory identifier job to be used on light rescans * Some optimizations on identifier and object validator jobs * merge jamie's identifier PR * fully repaired identifier job * properly hooked up object kind * inspector fix * fix video badge * small improvements to libraries settings * identifier and inspector improvements * fix feature flags and hook up context menu location utilities * BETTER CONTEXT MENU x100 * test-files * style tweaks * new icon designs * manifest * fix thumbnails on web * media data * New Location Watcher and some minor fixes * disable broken media_data extractor, wip * wip * function name fix * Fixing pnpm prep and some warnings * Solving a race condition beetween indexer job and FS event handlerSome other minor warnings * Generating thumbnails on watcher * Remove event handler on watcher * Some initial works on modify events and other small fixes * File update event * Trying to be more generic with used events and some tests to validate our assumptions * Turning on location metadata file * Introducing core unit tests on CI pipeline * Submiting new unit test assumptions to validate on windows CI * Fixing unit tests * Fixing unit tests again * Fixing unit tests * Fixing unit tests for macos * Fixing unit tests for macos again * New structure for platform dependent event handling Implementing event handlers for Linux and MacOS * minor fixes + rustfmt + clippy * Windows event handling * Introducing a feature gate to only use location watching on desktop app for now * Putting more stuff behind feature gates to avoid warnings * Adding feature to cargo test on CI * Changing some debug logs to trace logs and removing Jamie specific stuff * Make location removal from manager less async * fix build when "location-watcher" feature disabled * fix types + clippy * make location manager non-static * remove uses of `to_string_lossy` * more invalidate_query calls * Expose `library_ctx` directly to avoid needless clones * New materialized_path handling for directories * Removing cascade delete between file_path and object - Some other minor stuff * remove unused `CurrentDirFileIdentifierJob` Co-authored-by: Jamie Pine <ijamespine@me.com> Co-authored-by: Oscar Beaumont <oscar@otbeaumont.me>
2022-12-30 16:53:24 +00:00
let location_manager = LocationManager::new();
[ENG 239] Onboarding Flow & Location Settings (#529) * begin better onboarding * added input and altered text * better router & text + database icon Co-authored-by: maxichrome <maxichrome@users.noreply.github.com> * work on privacy screen + radio buttons * fix video extension bug and alter screens * add pending schema and location manager helper * functional onboarding * added secure temp store and started creating library loading screen * fix secure temp keystore + api * better onboarding * added location settings and some overview concept, all WIP * fix switch * prep * fix location router * added backend settings * attempted to fix form * begin indexer rules editor, plus tweaks * indexer rules coming soon * fix onboarding img size * cleanup * clone is needed here, but clippy no like * sike * whole bunch of fixes * clippy + ts * Removing some TODOs from api/libraries.rs and fixing db size calculation * moved object kind to client, added half functionality for appearance settings * fix RadioGroup helper * fix type issues * cargo fmt * fix creating library error handling + invalidate location list on update * forgot to switch back to onError * Invalidating getStatistics query on library creation and introducing the concept of waiting for a job on FileCopierJob * F* cargo fmt * fix RadioGroup interactivity * wipe all migrations * put back COLLATE NOCASE on extension columns * update core.ts * remove unused device component * fix typeerror in mobile --------- Co-authored-by: maxichrome <maxichrome@users.noreply.github.com> Co-authored-by: Brendan Allan <brendonovich@outlook.com> Co-authored-by: Ericson Soares <ericson.ds999@gmail.com> Co-authored-by: Utku Bakir <74243531+utkubakir@users.noreply.github.com>
2023-02-10 22:08:13 +00:00
let secure_temp_keystore = SecureTempKeystore::new();
let library_manager = LibraryManager::new(
data_dir.join("libraries"),
NodeContext {
config: Arc::clone(&config),
jobs: Arc::clone(&jobs),
ENG 223 Location Awareness (#468) * Introducing location online/offline checks and location relink * Some initial drafts for location watchers * Location metadata relink and add library * Many improvements at job system Now using prisma batching at identifier job Removing blocking I/O from extension subcrate Implementing lazy generation of thumbnails New current directory identifier job to be used on light rescans * Some optimizations on identifier and object validator jobs * merge jamie's identifier PR * fully repaired identifier job * properly hooked up object kind * inspector fix * fix video badge * small improvements to libraries settings * identifier and inspector improvements * fix feature flags and hook up context menu location utilities * BETTER CONTEXT MENU x100 * test-files * style tweaks * new icon designs * manifest * fix thumbnails on web * media data * New Location Watcher and some minor fixes * disable broken media_data extractor, wip * wip * function name fix * Fixing pnpm prep and some warnings * Solving a race condition beetween indexer job and FS event handlerSome other minor warnings * Generating thumbnails on watcher * Remove event handler on watcher * Some initial works on modify events and other small fixes * File update event * Trying to be more generic with used events and some tests to validate our assumptions * Turning on location metadata file * Introducing core unit tests on CI pipeline * Submiting new unit test assumptions to validate on windows CI * Fixing unit tests * Fixing unit tests again * Fixing unit tests * Fixing unit tests for macos * Fixing unit tests for macos again * New structure for platform dependent event handling Implementing event handlers for Linux and MacOS * minor fixes + rustfmt + clippy * Windows event handling * Introducing a feature gate to only use location watching on desktop app for now * Putting more stuff behind feature gates to avoid warnings * Adding feature to cargo test on CI * Changing some debug logs to trace logs and removing Jamie specific stuff * Make location removal from manager less async * fix build when "location-watcher" feature disabled * fix types + clippy * make location manager non-static * remove uses of `to_string_lossy` * more invalidate_query calls * Expose `library_ctx` directly to avoid needless clones * New materialized_path handling for directories * Removing cascade delete between file_path and object - Some other minor stuff * remove unused `CurrentDirFileIdentifierJob` Co-authored-by: Jamie Pine <ijamespine@me.com> Co-authored-by: Oscar Beaumont <oscar@otbeaumont.me>
2022-12-30 16:53:24 +00:00
location_manager: Arc::clone(&location_manager),
event_bus_tx: event_bus.0.clone(),
},
)
.await?;
ENG 223 Location Awareness (#468) * Introducing location online/offline checks and location relink * Some initial drafts for location watchers * Location metadata relink and add library * Many improvements at job system Now using prisma batching at identifier job Removing blocking I/O from extension subcrate Implementing lazy generation of thumbnails New current directory identifier job to be used on light rescans * Some optimizations on identifier and object validator jobs * merge jamie's identifier PR * fully repaired identifier job * properly hooked up object kind * inspector fix * fix video badge * small improvements to libraries settings * identifier and inspector improvements * fix feature flags and hook up context menu location utilities * BETTER CONTEXT MENU x100 * test-files * style tweaks * new icon designs * manifest * fix thumbnails on web * media data * New Location Watcher and some minor fixes * disable broken media_data extractor, wip * wip * function name fix * Fixing pnpm prep and some warnings * Solving a race condition beetween indexer job and FS event handlerSome other minor warnings * Generating thumbnails on watcher * Remove event handler on watcher * Some initial works on modify events and other small fixes * File update event * Trying to be more generic with used events and some tests to validate our assumptions * Turning on location metadata file * Introducing core unit tests on CI pipeline * Submiting new unit test assumptions to validate on windows CI * Fixing unit tests * Fixing unit tests again * Fixing unit tests * Fixing unit tests for macos * Fixing unit tests for macos again * New structure for platform dependent event handling Implementing event handlers for Linux and MacOS * minor fixes + rustfmt + clippy * Windows event handling * Introducing a feature gate to only use location watching on desktop app for now * Putting more stuff behind feature gates to avoid warnings * Adding feature to cargo test on CI * Changing some debug logs to trace logs and removing Jamie specific stuff * Make location removal from manager less async * fix build when "location-watcher" feature disabled * fix types + clippy * make location manager non-static * remove uses of `to_string_lossy` * more invalidate_query calls * Expose `library_ctx` directly to avoid needless clones * New materialized_path handling for directories * Removing cascade delete between file_path and object - Some other minor stuff * remove unused `CurrentDirFileIdentifierJob` Co-authored-by: Jamie Pine <ijamespine@me.com> Co-authored-by: Oscar Beaumont <oscar@otbeaumont.me>
2022-12-30 16:53:24 +00:00
// Adding already existing locations for location management
for library_ctx in library_manager.get_all_libraries_ctx().await {
for location in library_ctx
.db
.location()
.find_many(vec![])
.exec()
.await
.unwrap_or_else(|e| {
error!(
"Failed to get locations from database for location manager: {:#?}",
e
);
vec![]
}) {
if let Err(e) = location_manager.add(location.id, library_ctx.clone()).await {
error!("Failed to add location to location manager: {:#?}", e);
}
}
}
debug!("Watching locations");
// Trying to resume possible paused jobs
let inner_library_manager = Arc::clone(&library_manager);
let inner_jobs = Arc::clone(&jobs);
tokio::spawn(async move {
for library_ctx in inner_library_manager.get_all_libraries_ctx().await {
if let Err(e) = Arc::clone(&inner_jobs).resume_jobs(&library_ctx).await {
error!("Failed to resume jobs for library. {:#?}", e);
}
}
});
2022-05-23 04:07:35 +00:00
let router = api::mount();
2022-07-11 02:05:24 +00:00
let node = Node {
config,
library_manager,
2022-07-11 02:05:24 +00:00
jobs,
event_bus,
[ENG 239] Onboarding Flow & Location Settings (#529) * begin better onboarding * added input and altered text * better router & text + database icon Co-authored-by: maxichrome <maxichrome@users.noreply.github.com> * work on privacy screen + radio buttons * fix video extension bug and alter screens * add pending schema and location manager helper * functional onboarding * added secure temp store and started creating library loading screen * fix secure temp keystore + api * better onboarding * added location settings and some overview concept, all WIP * fix switch * prep * fix location router * added backend settings * attempted to fix form * begin indexer rules editor, plus tweaks * indexer rules coming soon * fix onboarding img size * cleanup * clone is needed here, but clippy no like * sike * whole bunch of fixes * clippy + ts * Removing some TODOs from api/libraries.rs and fixing db size calculation * moved object kind to client, added half functionality for appearance settings * fix RadioGroup helper * fix type issues * cargo fmt * fix creating library error handling + invalidate location list on update * forgot to switch back to onError * Invalidating getStatistics query on library creation and introducing the concept of waiting for a job on FileCopierJob * F* cargo fmt * fix RadioGroup interactivity * wipe all migrations * put back COLLATE NOCASE on extension columns * update core.ts * remove unused device component * fix typeerror in mobile --------- Co-authored-by: maxichrome <maxichrome@users.noreply.github.com> Co-authored-by: Brendan Allan <brendonovich@outlook.com> Co-authored-by: Ericson Soares <ericson.ds999@gmail.com> Co-authored-by: Utku Bakir <74243531+utkubakir@users.noreply.github.com>
2023-02-10 22:08:13 +00:00
secure_temp_keystore,
2022-05-23 04:07:35 +00:00
};
ENG 223 Location Awareness (#468) * Introducing location online/offline checks and location relink * Some initial drafts for location watchers * Location metadata relink and add library * Many improvements at job system Now using prisma batching at identifier job Removing blocking I/O from extension subcrate Implementing lazy generation of thumbnails New current directory identifier job to be used on light rescans * Some optimizations on identifier and object validator jobs * merge jamie's identifier PR * fully repaired identifier job * properly hooked up object kind * inspector fix * fix video badge * small improvements to libraries settings * identifier and inspector improvements * fix feature flags and hook up context menu location utilities * BETTER CONTEXT MENU x100 * test-files * style tweaks * new icon designs * manifest * fix thumbnails on web * media data * New Location Watcher and some minor fixes * disable broken media_data extractor, wip * wip * function name fix * Fixing pnpm prep and some warnings * Solving a race condition beetween indexer job and FS event handlerSome other minor warnings * Generating thumbnails on watcher * Remove event handler on watcher * Some initial works on modify events and other small fixes * File update event * Trying to be more generic with used events and some tests to validate our assumptions * Turning on location metadata file * Introducing core unit tests on CI pipeline * Submiting new unit test assumptions to validate on windows CI * Fixing unit tests * Fixing unit tests again * Fixing unit tests * Fixing unit tests for macos * Fixing unit tests for macos again * New structure for platform dependent event handling Implementing event handlers for Linux and MacOS * minor fixes + rustfmt + clippy * Windows event handling * Introducing a feature gate to only use location watching on desktop app for now * Putting more stuff behind feature gates to avoid warnings * Adding feature to cargo test on CI * Changing some debug logs to trace logs and removing Jamie specific stuff * Make location removal from manager less async * fix build when "location-watcher" feature disabled * fix types + clippy * make location manager non-static * remove uses of `to_string_lossy` * more invalidate_query calls * Expose `library_ctx` directly to avoid needless clones * New materialized_path handling for directories * Removing cascade delete between file_path and object - Some other minor stuff * remove unused `CurrentDirFileIdentifierJob` Co-authored-by: Jamie Pine <ijamespine@me.com> Co-authored-by: Oscar Beaumont <oscar@otbeaumont.me>
2022-12-30 16:53:24 +00:00
info!("Spacedrive online.");
Ok((Arc::new(node), router))
2022-05-23 04:07:35 +00:00
}
pub fn get_request_context(&self) -> Ctx {
Ctx {
library_manager: Arc::clone(&self.library_manager),
2022-07-11 21:21:50 +00:00
config: Arc::clone(&self.config),
jobs: Arc::clone(&self.jobs),
event_bus: self.event_bus.0.clone(),
[ENG 239] Onboarding Flow & Location Settings (#529) * begin better onboarding * added input and altered text * better router & text + database icon Co-authored-by: maxichrome <maxichrome@users.noreply.github.com> * work on privacy screen + radio buttons * fix video extension bug and alter screens * add pending schema and location manager helper * functional onboarding * added secure temp store and started creating library loading screen * fix secure temp keystore + api * better onboarding * added location settings and some overview concept, all WIP * fix switch * prep * fix location router * added backend settings * attempted to fix form * begin indexer rules editor, plus tweaks * indexer rules coming soon * fix onboarding img size * cleanup * clone is needed here, but clippy no like * sike * whole bunch of fixes * clippy + ts * Removing some TODOs from api/libraries.rs and fixing db size calculation * moved object kind to client, added half functionality for appearance settings * fix RadioGroup helper * fix type issues * cargo fmt * fix creating library error handling + invalidate location list on update * forgot to switch back to onError * Invalidating getStatistics query on library creation and introducing the concept of waiting for a job on FileCopierJob * F* cargo fmt * fix RadioGroup interactivity * wipe all migrations * put back COLLATE NOCASE on extension columns * update core.ts * remove unused device component * fix typeerror in mobile --------- Co-authored-by: maxichrome <maxichrome@users.noreply.github.com> Co-authored-by: Brendan Allan <brendonovich@outlook.com> Co-authored-by: Ericson Soares <ericson.ds999@gmail.com> Co-authored-by: Utku Bakir <74243531+utkubakir@users.noreply.github.com>
2023-02-10 22:08:13 +00:00
secure_temp_keystore: Arc::clone(&self.secure_temp_keystore),
2022-05-23 04:07:35 +00:00
}
}
pub async fn shutdown(&self) {
info!("Spacedrive shutting down...");
self.jobs.pause().await;
info!("Spacedrive Core shutdown successful!");
2022-05-23 04:07:35 +00:00
}
}
/// Error type for Node related errors.
#[derive(Error, Debug)]
pub enum NodeError {
#[error("Failed to create data directory: {0}")]
FailedToCreateDataDirectory(#[from] std::io::Error),
#[error("Failed to initialize config: {0}")]
FailedToInitializeConfig(#[from] node::NodeConfigError),
#[error("Failed to initialize library manager: {0}")]
FailedToInitializeLibraryManager(#[from] library::LibraryManagerError),
ENG 223 Location Awareness (#468) * Introducing location online/offline checks and location relink * Some initial drafts for location watchers * Location metadata relink and add library * Many improvements at job system Now using prisma batching at identifier job Removing blocking I/O from extension subcrate Implementing lazy generation of thumbnails New current directory identifier job to be used on light rescans * Some optimizations on identifier and object validator jobs * merge jamie's identifier PR * fully repaired identifier job * properly hooked up object kind * inspector fix * fix video badge * small improvements to libraries settings * identifier and inspector improvements * fix feature flags and hook up context menu location utilities * BETTER CONTEXT MENU x100 * test-files * style tweaks * new icon designs * manifest * fix thumbnails on web * media data * New Location Watcher and some minor fixes * disable broken media_data extractor, wip * wip * function name fix * Fixing pnpm prep and some warnings * Solving a race condition beetween indexer job and FS event handlerSome other minor warnings * Generating thumbnails on watcher * Remove event handler on watcher * Some initial works on modify events and other small fixes * File update event * Trying to be more generic with used events and some tests to validate our assumptions * Turning on location metadata file * Introducing core unit tests on CI pipeline * Submiting new unit test assumptions to validate on windows CI * Fixing unit tests * Fixing unit tests again * Fixing unit tests * Fixing unit tests for macos * Fixing unit tests for macos again * New structure for platform dependent event handling Implementing event handlers for Linux and MacOS * minor fixes + rustfmt + clippy * Windows event handling * Introducing a feature gate to only use location watching on desktop app for now * Putting more stuff behind feature gates to avoid warnings * Adding feature to cargo test on CI * Changing some debug logs to trace logs and removing Jamie specific stuff * Make location removal from manager less async * fix build when "location-watcher" feature disabled * fix types + clippy * make location manager non-static * remove uses of `to_string_lossy` * more invalidate_query calls * Expose `library_ctx` directly to avoid needless clones * New materialized_path handling for directories * Removing cascade delete between file_path and object - Some other minor stuff * remove unused `CurrentDirFileIdentifierJob` Co-authored-by: Jamie Pine <ijamespine@me.com> Co-authored-by: Oscar Beaumont <oscar@otbeaumont.me>
2022-12-30 16:53:24 +00:00
#[error("Location manager error: {0}")]
LocationManager(#[from] LocationManagerError),
}