Spacedrive is an open source cross-platform file explorer, powered by a virtual distributed filesystem written in Rust.
Go to file
Vítor Vasconcellos e797b02e65
Media metadata extraction & Thumbnailer rework (#2285)
* initial ffprobe commit

* Working slim down version ffprobe

* Auto format ffprobe and deps source

* Remove show_pixel_formats logic
- Fix do_bitexact incorrect check in main after last changes
- Fix some clangd warning

* Remove show_* and print_format options and their respective logic

* Rework ffprobe into simple_ffprobe
- Simplify ffprobe logic into a simple program that gather and print a media file metadata

* Reduce the amount of ffmpeg log messages while generating thumbnails

* Fix completly wrong comments

* mend

* Start modeling ffmpeg extracted metadata on schema
 - Start porting ffprobe code to rust
 - Rename some references to media_data to exif_data

* Finish modeling media info data
 - Add MediaProgram, MediaStream, MediaCodec, MediaVideoProps, MediaAudioProps, MediaSubtitleProps to Schema
 - Fix simple_ffproble to use its custom print_codec, instead of ffmpeg's impl

* Add relation between MediaInfo and FilePath
 - Remove shared properties from MediaInfo and related structs
 - Implement Iterator for FFmpegDict

* Fix and update schema

* Data models and start populating MediaInfo in rust

* Finish populating media info, chapters and program

* Improve FFmpegFormatContext data raw pointer access
 - Implement stream data gathering

* Impl FFmpegCodecContext, retrieve codec information
 - Improve some unsafe pointer uses
 - Impl from FFmpegFormatContext to MediaInfo conversion

* Fix FFmpegDict Drop

* Fix some crago warnings

* Impl retrieval of video props
 - Fix C char* to Rust String convertion

* Impl retrieval of audio and subtitle props
 - Fill props for MediaCodec

* Remove simple_ffprobe now that the Rust impl is done

* Fix schema to match actually retrieved media info
 - Fix import some FFmpeg constants instead of directly using values

* Rework movie_decoder
 - Re-implement create_scale_string and add support anamorphic video
 - Improve C pointer access for FFmpegFormatContext and FFmpegCodecContext
 - Use newer FFmpeg abstractions in movie_decoder

* Fix incorrect props when initializing MovieDecoder

* Remove unecessary lifetimes

* Added more native wrappers for some FFmpeg native objects used in movie_decoder

* Remove FFmpegPacket
 - Some more improvements to movie_decoder

* WIP

* Some small fixes

* More fixes
Rename movie_decoder to frame_decoder
Remove more references to film_strips

* fmt

* Fix duplicate migration for job error changes

* fix rebase

* Solving segfaults, fuck C lang

Co-authored-by: Vítor Vasconcellos <HeavenVolkoff@users.noreply.github.com>

* Update rust to version 1.77
 - Pin rust version with rust-toolchain.toml
 - Change from dtolnay/rust-toolchain to IronCoreLabs/rust-toolchain for rust-toolchain support
 - Remove unused function and imports
 - Replace most CString uses with new c literal string

* More segfault solving and other minor fixes

Co-authored-by: Vítor Vasconcellos <HeavenVolkoff@users.noreply.github.com>

* Fix ffmpeg rotation filter breaking portrait video thumbnails #2150
 - Plus some other misc fixes

* Auto format

* Retrieve video/audio metadata on frontend

* Auto format

* First draft on ffmpeg data save on db

Co-authored-by: Vítor Vasconcellos <HeavenVolkoff@users.noreply.github.com>

* Fix some incorrect changes to prisma schema

* Some fixes for the FFmpegData schema
 - Expand logic to save FFmpegData to db

* A ton of things

Co-authored-by: Vítor Vasconcellos <HeavenVolkoff@users.noreply.github.com>

* Integrating ffmpeg media data in jobs and API

* Rspc can't BigInt

* 🙄

* Add initial ffmpeg metadata entries to Inspector
 - Fix ephemeral metadata api to match the files metadata api call

* Fix Inspector not showing ffmpeg metadata

* Add bitrate, start time and chapters video metadata to Inspector
- Fix backend BigInt conversion incorrectly using i32 instead of u32
- Change FFmpegFormatContext/FFmpegMetaData bit_rate to i64
- Rename byteSize to humanizeSize
- Expand humanizeSize logic to allow handling bits and Binary units
- Move capitalize to @sd/client utils

* Solving some issues

* Fix ffmpeg probe getting incorrect stream id and breaking database unique constraint
 - Fix humanizeSize breaking when receiving floating numbers
 - Fix incorrect equality in StatCard
 - Fix unhandled error in Dialog when trying to remove an unknown dialog

* fmt

* small improvements
 - Remove some unecessary recursion_limit directive
 - Remove unused app_image releated functions
 - Fix metadata query enabled flag

* Add migration for ffmpeg media data

* Fix cypress test

* Requested changes

* Implement feedback
 - Update locale keys for all languages
 - Add pnpm command to update all language keys

* Fix thumb reactivity in non indexed locations

---------

Co-authored-by: Ericson Soares <ericson.ds999@gmail.com>
Co-authored-by: Vítor Vasconcellos <HeavenVolkoff@users.noreply.github.com>
2024-05-09 02:20:28 +00:00
.cargo Update github actions due to nodejs 16 deprecation (#2107) 2024-02-21 09:27:40 +00:00
.cspell Media metadata extraction & Thumbnailer rework (#2285) 2024-05-09 02:20:28 +00:00
.github Media metadata extraction & Thumbnailer rework (#2285) 2024-05-09 02:20:28 +00:00
.vscode Media metadata extraction & Thumbnailer rework (#2285) 2024-05-09 02:20:28 +00:00
apps Media metadata extraction & Thumbnailer rework (#2285) 2024-05-09 02:20:28 +00:00
core Media metadata extraction & Thumbnailer rework (#2285) 2024-05-09 02:20:28 +00:00
crates Media metadata extraction & Thumbnailer rework (#2285) 2024-05-09 02:20:28 +00:00
docs Improve prod bundle size (#2447) 2024-05-07 19:32:34 +00:00
extensions Linted Files according to the .prettierrc. 2022-04-24 21:16:55 +01:00
interface Media metadata extraction & Thumbnailer rework (#2285) 2024-05-09 02:20:28 +00:00
packages Media metadata extraction & Thumbnailer rework (#2285) 2024-05-09 02:20:28 +00:00
patches [ENG-1690] Upgrade to Tauri 2.0 (#2433) 2024-05-07 08:36:50 +00:00
scripts Improve prod bundle size (#2447) 2024-05-07 19:32:34 +00:00
.clippy.toml [ENG-835] Duplicating an empty directory doesn't duplicate nested empty directories (#1213) 2023-08-13 18:34:46 +00:00
.editorconfig [ENG-1479] AI Prototype (#1845) 2023-12-19 09:28:57 +00:00
.eslintignore Fix Windows CI + format (#767) 2023-04-27 07:32:15 +00:00
.gitattributes [ENG-469] Make Prettier and ESLint work together (#706) 2023-04-14 21:21:21 +00:00
.gitignore OpenDAL - Ephemeral Locations (#2283) 2024-04-12 06:51:51 +00:00
.npmrc Remove normalised cache (#2390) 2024-04-30 00:36:39 +00:00
.nvmrc Update npmrc (#2227) 2024-03-21 15:37:43 +00:00
.prettierignore Fix pnpm prep fails to download native-deps on some specific network env (#2275) 2024-04-04 14:19:49 +00:00
.prettierrc.js [ENG-1067] Update phosphor to new package & update sort imports (#1330) 2023-09-11 15:26:44 +00:00
.rustfmt.toml Added crates folder (#389) 2022-09-29 21:02:29 -07:00
Cargo.lock Media metadata extraction & Thumbnailer rework (#2285) 2024-05-09 02:20:28 +00:00
Cargo.toml Media metadata extraction & Thumbnailer rework (#2285) 2024-05-09 02:20:28 +00:00
CODE_OF_CONDUCT.md New team page (#229) 2022-06-14 13:13:09 -07:00
CONTRIBUTING.md Media metadata extraction & Thumbnailer rework (#2285) 2024-05-09 02:20:28 +00:00
cspell.config.yaml cleanup(don't hate me): prettier format everything 2023-01-19 19:32:35 -08:00
LICENSE Switch license to AGPL (#307) 2022-07-02 14:26:00 -07:00
package.json Media metadata extraction & Thumbnailer rework (#2285) 2024-05-09 02:20:28 +00:00
pnpm-lock.yaml [MOB-96] Haptics on tabs interaction and more (#2464) 2024-05-08 17:41:34 +00:00
pnpm-workspace.yaml Tauri 1.5.2 + normalize and update frontend dev deps (#1579) 2023-10-14 04:27:09 +00:00
README.md Update readme & contributing guide & language stuff (#2071) 2024-02-09 15:42:42 +00:00
rust-toolchain.toml Media metadata extraction & Thumbnailer rework (#2285) 2024-05-09 02:20:28 +00:00
SECURITY.md [ENG-1165] Add a security policy and notices to the appropriate READMEs (#1409) 2023-09-30 08:31:38 +00:00
tsconfig.json [ENG-380] Interface code structure improvement (#581) 2023-02-27 21:29:48 -08:00
turbo.json Improve prod bundle size (#2447) 2024-05-07 19:32:34 +00:00

Logo

Spacedrive

A file explorer from the future.
spacedrive.com »

Download for macOS (Apple Silicon | Intel) · Windows · Linux · iOS · Android
~ Links for iOS & Android will be added once a release is available. ~

Spacedrive is an open source cross-platform file manager, powered by a virtual distributed filesystem (VDFS) written in Rust.

UPDATE: Spacedrive is under active development, we are in the alpha stage and some features might be missing or disabled in the official release downloads.

Organize files across many devices in one place. From cloud services to offline hard drives, Spacedrive combines the storage capacity and processing power of your devices into one personal distributed cloud, that is both secure and intuitive to use.

For independent creatives, hoarders and those that want to own their digital footprint, Spacedrive provides a free file management experience like no other.

App screenshot


What is a VDFS?

A VDFS (virtual distributed filesystem) is a filesystem designed to work across a variety of storage layers. With a uniform API to manipulate and access content across many devices, VDFS is not restricted to a single machine. It achieves this by maintaining a virtual index of all storage locations, synchronizing the database between clients in realtime. This implementation also uses CAS (Content-addressable storage) to uniquely identify files, while keeping record of logical file paths relative to the storage locations.

The first implementation of a VDFS can be found in this UC Berkeley paper by Haoyuan Li. This paper describes its use for cloud computing, however the underlying concepts can be translated to open consumer software.

Motivation

Many of us have multiple cloud accounts, drives that arent backed up and data at risk of loss. We depend on cloud services like Google Photos and iCloud, but are locked in with limited capacity and almost zero interoperability between services and operating systems. Photo albums shouldnt be stuck in a device ecosystem, or harvested for advertising data. They should be OS agnostic, permanent and personally owned. Data we create is our legacy, that will long outlive us—open source technology is the only way to ensure we retain absolute control over the data that defines our lives, at unlimited scale.

Roadmap

View a list of our planned features here: spacedrive.com/roadmap

Developer Guide

Please refer to the contributing guide for how to install Spacedrive from sources.

Security Policy

Please refer to the security policy for details and information on how to responsibly report a security vulnerability or issue.

Architecture

This project is using what I'm calling the "PRRTT" stack (Prisma, Rust, React, TypeScript, Tauri).

  • Prisma on the front-end? 🤯 Made possible thanks to prisma-client-rust, developed by Brendonovich. Gives us access to the powerful migration CLI in development, along with the Prisma syntax for our schema. The application bundles with the Prisma query engine and codegen for a beautiful Rust API. Our lightweight migration runner is custom built for a desktop app context.
  • Tauri allows us to create a pure Rust native OS webview, without the overhead of your average Electron app. This brings the bundle size and average memory usage down dramatically. It also contributes to a more native feel, especially on macOS due to Safari's close integration with the OS.
  • We also use rspc which allows us to define functions in Rust and call them on the Typescript frontend in a completely typesafe manner, so no unnecessary bugs make it into production!
  • The core (sdcore) is written in pure Rust.

Monorepo structure:

Apps:

  • desktop: A Tauri app.
  • mobile: A React Native app.
  • web: A React webapp.
  • landing: A React app using Next.js.
  • server: A Rust server for the webapp. (planned)
  • cli: A Rust command line interface. (planned)
  • storybook: A React storybook for the UI components.

Core:

  • core: The Rust core, referred to internally as sdcore. Contains filesystem, database and networking logic. Can be deployed in a variety of host applications.
  • crates: Shared Rust libraries used by the core and other Rust applications.

Interface:

  • interface: The complete user interface in React (used by apps desktop, web)

Packages:

  • assets: Shared assets (images, fonts, etc).

  • client: A TypeScript client library to handle dataflow via RPC between UI and the Rust core.

  • config: eslint configurations (includes eslint-config-next, eslint-config-prettier and all tsconfig.json configs used throughout the monorepo).

  • ui: A React Shared component library.

  • macos: A Swift Native binary for MacOS system extensions (planned).

  • ios: A Swift Native binary (planned).

  • windows: A C# Native binary (planned).

  • android: A Kotlin Native binary (planned).