spacedrive/core/crates/sync
Ericson "Fogo" Soares bdc242a852
[ENG-1775] Scan location using new jobs (#2476)
* First draft on task system usage, still missing job system

* Scan location roughly working, a ton of stuff to fix yet

* Updating some deps due to crashes and bugs

* Exposing non critical errors to frontend

* Getting active job reports from job system

* Using boxed opaque type to avoid a downcast issue with generics

* Task system issues discovered on race conditions

* Enable debug

* Fix job report in the job manager

* Fix race condition on steal tasks

* Fixed race condition on task suspend

* Some fixes on job progress reporting and save

* Fixed many race conditions and a hard deadlock
Also some progress report polishing

* Ignore .ts and .mts video files for now

* Some better logs

* bruh

* Internal deadlocks and excess of communication in the task system
- Also better logs

* Bunch of fixes and optimizations

* WIP at fixing file identifier

* Fixed file identifier job
- still need to work on its progress report frontend

* A bunch of polishing

* Fixed some bugs and did more polishing

* Cleanup

* Bridging old and new job systems

* A ton of fixes

* A bunch of bugs related to shutdown and resume

* Indexer and watcher bugs

* Log normalizing

* Fixing CI

* Change error! to warn! on non critical errors log

* Fix redirect to new location

* Type annotation

* Bogus merge resolution on cargo lock
2024-06-17 21:30:57 +00:00
..
src [ENG-1775] Scan location using new jobs (#2476) 2024-06-17 21:30:57 +00:00
tests [ENG-1775] Scan location using new jobs (#2476) 2024-06-17 21:30:57 +00:00
Cargo.toml Clean up and update rust dependencies (#2544) 2024-06-10 19:44:00 +00:00
README.md [ENG-1776] Delete cloud ops after they've been ingested (#2512) 2024-05-31 06:54:22 +00:00

sd-core-sync

Spacedrive's sync system. Consumes types and helpers from sd-sync.

Creating Records

Prepare a sync id by creating or obtaining its value, and then wrapping it in the model's SyncId struct, available at prisma_sync::{model}::SyncId.

Next, prepare the sync operations using some varaints of the sync_entry macros. sync_entry and option_sync_entry take the value first, and then the path to the field's prisma module. sync_db_entry and option_sync_db_entry take the same inputs, but additionally produce a prisma operation in a tuple with the sync operation, intended to be put into a Vec and unzipped.

Finally, use sync.shared/relation_create depending on if you're creating a standalone record or a relation between two records, and then write it to the database with write_ops.

let (sync_params, db_params): (Vec<_>, Vec<_>) = [
  sync_db_entry!(self.name, tag::name),
  sync_db_entry!(self.color, tag::color),
  sync_db_entry!(false, tag::is_hidden),
  sync_db_entry!(date_created, tag::date_created),
]
.into_iter()
.unzip();

sync.write_ops(
  db,
  (
    sync.shared_create(
      prisma_sync::tag::SyncId { pub_id },
      sync_params,
    ),
    db.tag().create(pub_id, db_params),
  ),
)

Updating Records

This follows a similar process to creation, but with sync.shared/relation_create.

let (sync_params, db_params): (Vec<_>, Vec<_>) = [
  sync_db_entry!(name, tag::name),
  sync_db_entry!(color, tag::color),
]
.into_iter()
.unzip();

sync.write_ops(
  db,
  (
    sync.shared_update(prisma_sync::tag::SyncId { pub_id }, k, v),
    db.tag().update(tag::id::equals(id), db_params);
  )
)

Deleting Records

This only requires a sync ID.

sync.write_op(
  db,
  sync.shared_delete(prisma_sync::tag::SyncId { pub_id }),
  db.tag().delete(tag::id::equals(id));
)

Relation Records

Relations require sync IDs for both the item and the group being related together. Apart from that they're basically the same as shared operations.

let (sync_params, db_params): (Vec<_>, Vec<_>) = [
  sync_db_entry!(date_created, tag_on_object::date_created)
]
.into_iter()
.unzip();

sync.write_ops(
  db,
  (
    sync.relation_create(
      prisma_sync::tag_on_object::SyncId {
        tag: prisma_sync::tag::SyncId { pub_id: tag_pub_id },
        object: prisma_sync::object::SyncId { pub_id: object_pub_id },
      },
      sync_params
    ),
    db.tag_on_object().create(
        object::id::equals(object_id),
        tag::id::equals(tag_id),
        db_params
    )
  )
)

Setting Relation Fields

Setting relation fields requires providing the Sync ID of the relation. Setting the relation field's scalar fields instead will not properly sync then relation, usually because the scalar fields are local and disconnected from the Sync ID.

let (sync_params, db_params): (Vec<_>, Vec<_>) = [
	sync_db_entry!(
		prisma_sync::object::SyncId { pub_id: object_pub_id },
		file_path::object
	)
].into_iter().unzip();

sync.write_ops(
	db,
	(
		sync.shared_update(
			prisma_sync::file_path::SyncId {
				pub_id: file_path_pub_id
			},
			sync_params
		),
		db.file_path().update(
			file_path::id::equals(file_path_id),
			db_params
		)
	)
)