The pixel buffer pool is used to create pixel buffers with both the CMIO
extension and the DAL plugin. As such, it is created independent of
which camera type is used, and should be released independent of it as
well.
This deprecates the following functions, replacing them with new
versions:
- `obs_output_can_begin_data_capture()` - now `*capture2()`
- `obs_output_initialize_encoders()` - now `*encoders2()`
- `obs_output_begin_data_capture()` - now `*capture2()`
The flags parameter was initially designed to support audio-only or
video-only operation of an output which had the `OBS_OUTPUT_AV` flag,
however, full support for that was never implemented, and there are
likely fundamental issues with an implementation, mainly that most
outputs are programmed assuming that there will always be at least one
audio and one video track. This requires new flags specifying support
for optional audio/video, among other things.
An implementation to allow audio/video to be optional is best done
using the flag technique above, with audio/video enablement specified
by whether media (raw, `video_t/audio_t`) or encoder (`obs_encoder_t`)
objects are specified.
Since every implementation I could find always specifies `flags` as 0,
I was able to safely conclude that immediately removing the parameter's
functionality is safe to do.
Camera Extensions require specific entitlements for the hosting app,
which also require a provisioning profile. To avoid breaking local
builds that do not require the camera extension, an additional
entitlements file that will not trigger the provisioning profile
requirement will be used if the virtualcam (but not the Camera
Extension) is configured.
With the recent change to enforce C17 standard for compilers, default
visibility was also enforced to be hidden. This also automatically hid
the `PlugInMain` symbol required for DAL plugins. Adding the
`__exported` decorator makes the symbol explicitly visible.
All presentation time stamps are rounded to whole seconds during the
conversion from nanoseconds to seconds, because of the immediate cast
to `int64_t`. This results in the same presentation time stamp being
send to consumers for a whole second.
In previews/live streams this isn't super visible as last frame is
often assumed to be the newest and the stream is updated. It's more
problematic when recording since APIs like Apple's AVFoundation don't
allow duplicate presentation time stamps or it can look like frames are
produced in huge bursts once per second.
In this PR `CMTimeMakeWithSeconds` is used instead of `CMTimeMake` to
make sure the conversion is done correctly and simplify the calculation
we have to do a little.
The `fillFrame` method of the mac-virtualcam plugin is creating samples
directly using the value from `mach_absolute_time()` as `hostTime`.
This assumes this value is in nanoseconds, while it is in fact in mach
tick units. On Intel Macs mach tick units will be exactly 1 nanosecond
resulting in valid values, but on Apple Silicon macs this is no longer
the case.
This results in sample buffers with the placeholder image that have
much lower presentation time stamps than the samples containing content
produced by OBS. In previews/live streams this shows due to the last
content sample being shown frozen until the whole buffer is filled with
samples containing the placeholder image. Applications recording the
video stream are even more confused and crash or record videos with
wildly incorrect lengths.
In this PR `clock_gettime_nsec_np` is used to convert from mach tick units
to nanoseconds. This will make sure the `hostTime` value is correct on
both Apple Silicon and Intel macs. Making sure we produce stable
presentation time stamps from the virtual camera plugin at all times.
CMake build framework 3.0 introduces more strict typing checks, which
require additional code fixes to silence.
* Fixes warnings about implicit casts
* Fixes warnings about weak references used multiple times inside
blocks
The DAL plugin-based virtualcamera shares data between OBS and the
plugin using an IOSurface. IOSurface locks are necessary to ensure
race conditions between data generation (OBS side) and consumption
(virtual camera side) and also that an IOSurface is not offloaded to
GPU memory when it is indeed needed in CPU memory.
Also moves the invalidation of the NSMachPort for the frames to after
the IOSurface data has been converted into a pixelbuffer and added to
the frame queue of the virtual camera, as an early invalidation will
cut off access to the pixel data shared with the DAL plugin.
IOSurface locks are only necessary when any processing with the data
contained in the surface is done and an explicit copy of data from GPU
memory back to CPU memory is needed.
Apple Silicon-based Macs have a unified memory architecture, as such
an IOSurface will always be available in memory accessible to the CPU
and GPU (and an off-load of the IOSurface will not take place).
eGPUs are not supported on Apple Silicon-based Macs either, so an
IOSurface lock to ensure data is copied back to CPU memory is not
necessary.
Without invalidating the mach port used for sharing the IOSurface
between OBS and the application displaying the virtual camera output,
IOKit seems to run into the issue of receiving "shared" mach ports,
possibly because of port exhaustion. IOKit requires a "new" port
however and crashes upon that error otherwise.
Co-authored-by: Steven Michaud <smichaud@pobox.com>
The root cause of the issue is `swscale` dropping the second plane of
biplanar pixel data, resulting in an "incomplete" frame being fed to the
CVPixelBuffer.
As CVPixelBuffers have dedicated support for full range colour, use
these directly, which improves performance even further (as any
conversion for full range data is avoided as well).
To ensure that OBS does not implicitly enable conversion via `swscale`
a video conversion struct needs to be set in any case, ensuring that the
output range and colourspace match the output configuration.
This issue primarily seems to affect Intel-based Macs without unified
memory: While an IOSurface is supposed to be shared via a mach port
with other processes, each process needs to lock the surface during
access (as is common with other shared resources).
Apple Silicon-based Macs seem to be less affected as a switch between
GPU and CPU memory (which can happen dynamically for IOSurfaces) would
still point to the same unified memory.
Converting the color space was likely accidental as it isn't explicitly
set. Unfortunately, not setting it means that it gets set to the
default, which is Rec. 709 and thus a conversion takes place when having
any other space. This conversion leads to a massive performance penalty
that isn't necessary.
This change fixes a memory leak in the mac-virtualcam plugin that causes
OBS to not release the CVPixelBuffers (and underlying IOSurfaces)
it emits to the virtual camera consumers.
Pull request https://github.com/obsproject/obs-studio/pull/6573 (Avoid
transcoding where possible) updated the mac-virtualcam to share the
virtual camera feed with other processes via IOSurfaces.
Although the changes work correctly, users have observed that OBS memory
usage keeps increasing when the virtual camera is active until OBS runs
out of memory or the consuming application is closed.
See the report by @SciTechNick for more information:
https://github.com/obsproject/obs-studio/pull/6573#issuecomment-1161979765
After some debugging, I have found that the plugin is leaking Mach ports
associated with IOSurfaces, preventing them from being re-used. The
previous approach using `NSMachPort` does not seem to properly release
the Mach port allocated via `CVPixelBufferGetIOSurface` and
`IOSurfaceLookupFromMachPort`. Instead, we must explicitly deallocate
the port using `mach_port_deallocate`.
I have tested the changes on a Macbook Pro (M1) running macOS Monterey with
Google Chrome, Zoom, and Cameo. OBS shows no signs of memory leakage
after multiple minutes.
This change fixes an issue with the CMIO DAL plugin where the CMIO
subsystem would log multiple errors when starting the virtual camera,
due to certain properties that could not be set (frame rate and format).
For now, we just ignore the assignment, but mark the property as
settable to suppress the error messages that are reported by the CMIO
subsystem.
This change fixes an issue in the Mach server exposed by the macOS
virtual camera for OBS, where it would not invalidate ports that were
disconnected by the remote application, causing sporadic crashes.
These crashes can be reproduced in the previous builds by opening the
virtual camera in a remote application and closing the application
(without stopping the virtual camera).
This change updates the implementation of the mac-virtualcam plugin to
not use any global state and instead rely on the state object that is
passed by the OBS module system.
This approach is similar to the virtual camera implementations for Linux
and Windows.
This change removes the unused CMSampleBuffer utility functions that
were still left from the previous implementation. Since we construct the
CMSampleBuffer directly from an IOSurface, we do not need any custom
construction logic anymore, since that is now performed by the OBS
plugin.
This change updates the mac-virtualcam implementation to conditionally
enable conversion of the output video format. Previously, the output
video was always converted into UYVY. However, this conversion exhibits
high CPU usage, as reported in:
https://github.com/johnboiles/obs-mac-virtualcam/issues/102
Therefore, we disable conversion when the selected output format (e.g.,
NV12) is natively supported by CoreVideo's pixel buffers.
This change updates the plugin to support video formats that contain
multiple planes (such as NV12). Such functionality is necessary to
prevent transcoding the raw video data, which is often delivered in a
planar format.
This change updates the mac-virtualcam implementation to pool the
CVPixelBuffers used to share the output frames. This allows the plugin
to recycle the pixel buffers used by the plugin.
This change updates the virtual camera implementation on macOS to
utilize IOSurface to share the output feed with the virtual cameras.
By using IOSurface, we remove the need for copying the frames across
multiple buffers, since they can be shared across Mach connections using
zero-copy.
This change fixes an issue where the DAL plugin would not load due to
not supporting the architecture arm64e. We update the build
configuration to build a universal binary that includes arm64e as well.
See https://github.com/obsproject/obs-studio/issues/6285 for more
information regarding this issue.
Status output related to OBS configuration is prefixed with the string
"OBS" and added padding for enabled and disabled features. This padding
was not aligned between platforms.
By moving the padding and prefix decoration into its own function,
both elements are controlled in a single place. CMake scripts were
changed to use this new function `obs_status` instead of using CMake's
`message` function directly.
