spacedrive/CONTRIBUTING.md
Vítor Vasconcellos e797b02e65
Media metadata extraction & Thumbnailer rework (#2285)
* initial ffprobe commit

* Working slim down version ffprobe

* Auto format ffprobe and deps source

* Remove show_pixel_formats logic
- Fix do_bitexact incorrect check in main after last changes
- Fix some clangd warning

* Remove show_* and print_format options and their respective logic

* Rework ffprobe into simple_ffprobe
- Simplify ffprobe logic into a simple program that gather and print a media file metadata

* Reduce the amount of ffmpeg log messages while generating thumbnails

* Fix completly wrong comments

* mend

* Start modeling ffmpeg extracted metadata on schema
 - Start porting ffprobe code to rust
 - Rename some references to media_data to exif_data

* Finish modeling media info data
 - Add MediaProgram, MediaStream, MediaCodec, MediaVideoProps, MediaAudioProps, MediaSubtitleProps to Schema
 - Fix simple_ffproble to use its custom print_codec, instead of ffmpeg's impl

* Add relation between MediaInfo and FilePath
 - Remove shared properties from MediaInfo and related structs
 - Implement Iterator for FFmpegDict

* Fix and update schema

* Data models and start populating MediaInfo in rust

* Finish populating media info, chapters and program

* Improve FFmpegFormatContext data raw pointer access
 - Implement stream data gathering

* Impl FFmpegCodecContext, retrieve codec information
 - Improve some unsafe pointer uses
 - Impl from FFmpegFormatContext to MediaInfo conversion

* Fix FFmpegDict Drop

* Fix some crago warnings

* Impl retrieval of video props
 - Fix C char* to Rust String convertion

* Impl retrieval of audio and subtitle props
 - Fill props for MediaCodec

* Remove simple_ffprobe now that the Rust impl is done

* Fix schema to match actually retrieved media info
 - Fix import some FFmpeg constants instead of directly using values

* Rework movie_decoder
 - Re-implement create_scale_string and add support anamorphic video
 - Improve C pointer access for FFmpegFormatContext and FFmpegCodecContext
 - Use newer FFmpeg abstractions in movie_decoder

* Fix incorrect props when initializing MovieDecoder

* Remove unecessary lifetimes

* Added more native wrappers for some FFmpeg native objects used in movie_decoder

* Remove FFmpegPacket
 - Some more improvements to movie_decoder

* WIP

* Some small fixes

* More fixes
Rename movie_decoder to frame_decoder
Remove more references to film_strips

* fmt

* Fix duplicate migration for job error changes

* fix rebase

* Solving segfaults, fuck C lang

Co-authored-by: Vítor Vasconcellos <HeavenVolkoff@users.noreply.github.com>

* Update rust to version 1.77
 - Pin rust version with rust-toolchain.toml
 - Change from dtolnay/rust-toolchain to IronCoreLabs/rust-toolchain for rust-toolchain support
 - Remove unused function and imports
 - Replace most CString uses with new c literal string

* More segfault solving and other minor fixes

Co-authored-by: Vítor Vasconcellos <HeavenVolkoff@users.noreply.github.com>

* Fix ffmpeg rotation filter breaking portrait video thumbnails #2150
 - Plus some other misc fixes

* Auto format

* Retrieve video/audio metadata on frontend

* Auto format

* First draft on ffmpeg data save on db

Co-authored-by: Vítor Vasconcellos <HeavenVolkoff@users.noreply.github.com>

* Fix some incorrect changes to prisma schema

* Some fixes for the FFmpegData schema
 - Expand logic to save FFmpegData to db

* A ton of things

Co-authored-by: Vítor Vasconcellos <HeavenVolkoff@users.noreply.github.com>

* Integrating ffmpeg media data in jobs and API

* Rspc can't BigInt

* 🙄

* Add initial ffmpeg metadata entries to Inspector
 - Fix ephemeral metadata api to match the files metadata api call

* Fix Inspector not showing ffmpeg metadata

* Add bitrate, start time and chapters video metadata to Inspector
- Fix backend BigInt conversion incorrectly using i32 instead of u32
- Change FFmpegFormatContext/FFmpegMetaData bit_rate to i64
- Rename byteSize to humanizeSize
- Expand humanizeSize logic to allow handling bits and Binary units
- Move capitalize to @sd/client utils

* Solving some issues

* Fix ffmpeg probe getting incorrect stream id and breaking database unique constraint
 - Fix humanizeSize breaking when receiving floating numbers
 - Fix incorrect equality in StatCard
 - Fix unhandled error in Dialog when trying to remove an unknown dialog

* fmt

* small improvements
 - Remove some unecessary recursion_limit directive
 - Remove unused app_image releated functions
 - Fix metadata query enabled flag

* Add migration for ffmpeg media data

* Fix cypress test

* Requested changes

* Implement feedback
 - Update locale keys for all languages
 - Add pnpm command to update all language keys

* Fix thumb reactivity in non indexed locations

---------

Co-authored-by: Ericson Soares <ericson.ds999@gmail.com>
Co-authored-by: Vítor Vasconcellos <HeavenVolkoff@users.noreply.github.com>
2024-05-09 02:20:28 +00:00

10 KiB

Welcome to the Spacedrive Contributing Guide

Thank you for investing your time in contributing to our project!

Please read our Code of Conduct to keep our community approachable and respectable.

This guide will provide an overview of the contribution workflow, including opening an issue, creating a pull request (PR), and the review and merge process.

New Contributor Guide

To familiarize yourself with the project, please read the README. Here are some resources to help you get started with open-source contributions:

Getting Started

Issues

Creating a New Issue

If you come across an issue or have a feature request for Spacedrive, please search if a related issue has already been reported. If no relevant issue exists, you can open a new issue using the appropriate issue form.

Solving an Issue

To find an issue that interests you, you can browse through our existing issues and use the available labels to narrow down your search (See Labels for more information). As a general rule, if you find an issue you want to work on, you are welcome to open a PR with a fix.

Making Changes

Making Changes Locally

This project uses Cargo and pnpm. Make sure you have them installed before proceeding.

To make changes locally, follow these steps:

  1. Clone the repository: git clone https://github.com/spacedriveapp/spacedrive
  2. Navigate to the project directory: cd spacedrive
  3. Configure your system environment for Spacedrive development
    1. For Linux users, run: ./scripts/setup.sh

      This script will check if Rust and pnpm are installed then proceed to install Clang, NASM, LLVM, libvips, Gstreamer's Plugins, FFmpeg, Perl, Tauri essentials and any other required dependencies for Spacedrive to build.

    2. For macOS users, run: ./scripts/setup.sh

      This script will check if Rust, pnpm and Xcode are installed and proceed to use Homebrew to install NASM, Tauri essentials and install any other required dependencies for Spacedrive to build.

    3. For Windows users, run in PowerShell: .\scripts\setup.ps1

      This script will install pnpm, LLVM, FFmpeg, C++ build tools, NASM, Rust + Cargo, Rust tools, Edge Webview 2, Strawberry Perl, Tauri essentials and any other required dependencies for Spacedrive to build.

  4. Install dependencies: pnpm i
  5. Prepare the build: pnpm prep (This will run all necessary codegen and build required dependencies)

To quickly run only the desktop app after prep, you can use:

  • pnpm tauri dev

    If necessary, the webview devtools can be opened by pressing Ctrl + Shift + I (Linux and Windows) or Command + Option + I (macOS) in the desktop app.

    Also, the react-devtools can be launched using pnpm dlx react-devtools. However, it must be executed before starting the desktop app for it to connect.

You can download a bundle with sample files to test the app by running:

  • pnpm test-data

    Only for Linux and macOS (Requires curl and tar). The test files will be located in a directory called test-data in the root of the spacedrive repo.

To run the web app:

  • pnpm dev:web

This will start both the server and web interface. You can launch these individually if you'd prefer:

  • cargo run -p sd-server (server)
  • pnpm web dev (web interface)

To run the e2e tests for the web app:

  • pnpm web test:e2e

If you are developing a new test, you can execute Cypress in interactive mode with:

  • pnpm web test:interactive

To run the landing page:

  • pnpm landing dev

If you encounter any issues, ensure that you are using the following versions of Rust, Node and Pnpm:

  • Rust version: 1.78
  • Node version: 18.18
  • Pnpm version: 9.0.6

After cleaning out your build artifacts using pnpm clean, git clean, or cargo clean, it is necessary to re-run the setup-system script.

Make sure to read the guidelines to ensure that your code follows a similar style to ours.

Mobile App

To run the mobile app:

  • Install Java JDK <= 17 for Android
  • Install Android Studio for Android and Xcode for iOS development.
  • Run ./scripts/setup.sh mobile
    • This will set up most of the dependencies required to build the mobile app.
  • Make sure you have NDK 26.1.10909125 and CMake installed in Android Studio.
  • Run the following commands:
    • pnpm mobile android (runs on Android Emulator)
      • In order to have locations working on Android, you must run the following command once the application has been installed for the first time. Otherwise, locations will not work.
        • adb shell appops set --uid com.spacedrive.app MANAGE_EXTERNAL_STORAGE allow
      • Run the following commands to access the logs from sd-core.
        • adb shell
        • Then run-as com.spacedrive.app to access the app's directory on device.
        • Run cd files/logs and then select the logs with the timestamp of when you ran the app. Ex: sd.log.2023-11-28.
          • You can view the logs using tail -f [log-name]. Ex: tail -f sd.log.2023-11-28.
    • pnpm mobile ios (runs on iOS Emulator)
      • xcrun simctl launch --console booted com.spacedrive.app allows you to view the console output of the iOS app from tracing. However, the application must be built in debug mode for this.
    • pnpm mobile start (runs the metro bundler only)

Pull Request

Once you have finished making your changes, create a pull request (PR) to submit them.

  • Fill out the "Ready for review" template to help reviewers understand your changes and the purpose of your PR.
  • If you are addressing an existing issue, don't forget to link your PR to the issue.
  • Enable the checkbox to allow maintainer edits so that the branch can be updated for merging.
  • Once you submit your PR, a team member will review your proposal. They may ask questions or request additional information.
  • You may be asked to make changes before the PR can be merged, either through suggested changes or pull request comments. You can apply suggested changes directly through the UI. For other changes, you can make them in your fork and commit them to your branch.
  • As you update your PR and apply changes, mark each conversation as resolved.
  • If you run into any merge issues, refer to this git tutorial to help you resolve merge conflicts and other issues.

Your PR is Merged!

Congratulations! 🎉🎉 The Spacedrive team thanks you for your contribution!

Once your PR is merged, your changes will be included in the next release of the application.

Common Errors

xcrun: error: unable to find utility "xctest", not a developer tool or in PATH

This error occurs when Xcode is not installed or when the Xcode command line tools are not in your PATH.

To resolve this issue:

  • Install Xcode from the macOS App Store or directly from here (requires Apple Account).
  • Run xcode-select -s /Applications/Xcode.app/Contents/Developer. This command will use Xcode's developer tools instead of macOS's default tools.

unable to lookup item 'PlatformPath'

If you run into this issue, or similar:

error: terminated(1): /us/bin/xcrun --sdk macos --show-sdk-platform-path output :
xcrun: error: unable to lookup item 'PlatformPath' from command line tools installation xcrun: error: unable to lookup item 'PlatformPath' in SDK '/Library/Developer /CommandLineTools/SDKs/MacOSX.sdk'

Ensure that macOS is fully updated, and that you have Xcode installed (via the app store).

Once that has completed, run xcode-select --install in the terminal to install the command line tools. If they are already installed, ensure that you update macOS to the latest version available.

Also ensure that Rosetta is installed, as a few of our dependencies require it. You can install Rosetta with softwareupdate --install-rosetta --agree-to-license.

Translations

Check out the i18n README for more information on how to contribute to translations.

Credits

This CONTRIBUTING.md file was inspired by the github/docs CONTRIBUTING.md file, and we extend our gratitude to the original author.