Eventually, most things should be replaced with Load where applicable
(though in some cases sub-pixel sampling is desired).
This commit also fixes a bug where NV12 async sources wouldn't render
correctly.
(Note: This commits also modifies the linux-pulseaudio, mac-capture, and
win-wasapi plugins)
Do not prevent the targeted output device from being monitored if the
selected monitor output device is a different one.
Closesjp9000/obs-studio#872
This change prevents source flags from being incorrectly overwritten and
set to 0. Eventually flags need to be separated from source settings
and this should be reverted, but for now this solves an issue where
buffering would be enabled on async video sources regardless of whether
the user disabled it or not on the source.
Adds functions to turn on audio monitoring to allow the user to hear
playback of an audio source over the user's speaker. It can be set to
turn off monitoring and only output to stream, or it can be set to
output only to monitoring, or it can be set to both.
On windows, audio monitoring uses WASAPI. Windows also is capable of
syncing the audio to the video according to when the video frame itself
was played.
On mac, it uses AudioQueue.
On linux, it's not currently implemented and won't do anything (to be
implemented).
Fixes a bug that would allow possible infinite recursion within a source
tree. To fix this, inactive sources must be enumerated as well in order
to prevent infinite recursion.
Commit 53955301a2 introduced a async source texture copy bug due to
creating a new case in a switch without adding a break to the one above
it, causing it to execute both cases by mistake.
Because D3D11 specifically does not support an L8 texture format (you
have to use a shader swizzle), manually convert Y800 signals to RGBX
instead. This also fixes a bug where Y800 signals will render red.
Closesjp9000/obs-studio#718
If an async source is cropped on one side, then when the program is
restarted and the source is loaded from file, the async source will
start out with a width/height of zero. This will cause the async source
to not be drawn if cropping or scale filtering is added to the scene
item, because it has to be rendered to a texture first. However, the
source cannot reset its size until it's drawn, so it leaves it in
perpetual state of having a 0x0 size.
This fixes that problem by ensuring that the async source size is always
reset even when not being rendered.
Closejp9000/obs-studio#686
When a scene is duplicated the filters on the scene were not copied to
the new scene. This causes that a temporary copy of a scene renders
differently in the program than in the preview when using studio mode.
(Note: This commit also modifies coreaudio-encoder, win-capture, and
win-mf modules)
This reduces logging to the user's log file. Most of the things
specified are not useful for examining log files, and make reading log
files more painful.
The things that are useful to log should be up to the front-end to
implement. The core and core plugins should have minimal mandatory
logging.
The active_refs and show_refs variable would only increment/decrement
their children if their values were 1 and 0, which means that in the
case of scenes within scenes, sub-sources of scenes within scenes would
end up having the wrong ref values.
When using GPU conversion for 4:2:0 frames on async video sources, it
would create a texture bigger than necessary and try to copy too much
data from the frame, resulting in a crash.
When a transition is a sub-source of another source, it would not call
the transition's active source enum function, meaning that any sources
the transition had would not increment their active/showing refs (it
would only be called when activating the transition directly before).
That would result in negative/invalid active/showing refs on its
sub-sources, causing them to become permanently active/inactive and/or
permanently showing/hidden.
(Note: this commit also modifies the obs-filters and test-input modules)
Changes the obs_source_process_filter_begin return type so that it
returns true/false to indicate that filter processing should or should
not continue (for example if the filter is bypassed or if there's some
other sort of issue that causes the filtering to fail)
The source shouldn't be inserted into obs->data.first_audio_source until it's
fully initialized, or other threads will access source->control and
dereference an uninitialized pointer.
This patch fixes a specific crash where if the user named a filter the
same name as an input source that already existed in the system, scene
item loading code could find the filter with the same name instead of
the source, and mistakenly use it as the scene item's source directly.
This would cause a crash when trying to render that filter as a regular
source.
Marking filters as private is a temporary and simple workaround to the
solution. Filters are currently not meant to be found via the main
enumeration/search functions, which is a design flaw (lack of
consistency). In future major API revisions of libobs, filters should
be reworked to act as sources, with the sources they filter as
sub-sources ideally.
Additionally, the concept of "private context objects" and "primary
lists of context objects" in the back-end should probably also be
removed, allowing the font-end (or optional separate API layers) to
control all primary lists of obs context objects. These minor issues
that occur ultimately stem from API design flaws which need to be
corrected.
This crash happened when a filter was mistakenly used as a regular
source due to an unrelated bug in filter code and scene loading code.
The filter and the source it belongs to both had the same names, and the
source loading code found the filter and mistakenly used it as the
source instead of the actual source with the same name.
(Note: Also modified the obs-ffmpeg plugin module)
Allows the ability for frame data to pass 8-bit grayscale images (Y800
color format).
Closesjp9000/obs-studio#515
Adds deinterlacing API functions. Both standard and 2x variants are
supported. Deinterlacing is set via obs_source_set_deinterlace_mode and
obs_source_set_deinterlace_field_order.
This was implemented in to the core itself because deinterlacing should
happen before effect filters are processed, but after async filters are
processed. If this were added as a filter, there is the possibility
that a different filter is processed before deinterlacing, which could
mess with the result. It was also a bit easier to implement this way
due to the fact that that deinterlacing may need to have access to the
previous async frame.
Effects were split in to separate files to reduce load time (especially
for yadif shaders which take a significant amount of time to compile).
Instead of just updating the async texture variables directly in the
source, allow the ability to pass the async texture variables via
function parameters to allow the ability to parse more than one frame to
more than one texture.
This code is primarily intended to be used to upload/convert the
"previous" async frame for the deinterlacer (if necessary).
(Note: This commit also modifies obs-filters and text-freetype2)
This simplifies writing of effects. DrawMatrix is no longer necessary
because there are no sources that require drawing with a color matrix
other than async sources, and async sources are automatically processed
and don't defer their initial render stage to filters.
Under certain circumstances, the timing_adjust variable would cause line
1161 to continually trigger over and over again. The "loop detection"
code incorrectly made it so that any timestamp that was just simply
below the expected value would be seen as a jump. After that, the
timing_adjust variable would be set for the frame again, and then the
audio would see it as a jump again after that, and those two things
would continue endlessly. This would cause stuttering particularly with
certain devices (particularly elgato/lgp/hdpvr) where the audio/video
data are decoded and sent at varying/different/unpredictable times.
To fix this issue, it should not detect values below as jumps, but
instead should only do it for values that exceed the MAX_TS_VAR (maximum
timestamp variance) value.
If obs_source::audio_ts is set to 0 (such as by discard_if_stopped in
obs-audio.c), but the push_back variable in the source_output_audio_data
function in obs-source.c was being set to true (meaning it's within the
seamless audio smoothing threshold), it would cause it to never reset
the obs_source::audio_ts value, and thus all audio data from the source
would become perpetually ignored by the audio subsystem until there was
finally some sort of timestamp jump that caused it to call
source_output_audio_place, and thus reset obs_source::audio_ts.
obs_source::audio_ts is only reset in source_output_audio_place, not in
source_output_audio_push_back, so the most simple solution is to just
call source_output_audio_push_back is obs_source::audio_ts is 0.
There's technically no need to clear the audio data here, nor is there
any need to try to trick the timestamp in to a different position. It
can simple just reset the audio timing.
Prevents a possible case where audio data might be deleted when it's not
necessary to delete any.
This variable is used to detect whether audio has stopped -- if audio
stops, it detects that no new data is coming in, and resets the audio
position so that it eliminates the chance of causing the audio buffering
to go haywire if audio starts up again. However, this variable was not
being reset every time the value changes, which it should.
The seamless audio looping code would erroneously trigger for things
that weren't loops, causing the audio data to continually push back and
ignore timestamps, thus going out of sync.
There does need to be loop handling code, but due to the fact that other
things may need to trigger this code, it's best just to clear the audio
data and start from a fresh sync point. Unfortunately for the case of
loops, this means the window in which audio data loops and video frames
loop need to be muted.
This fixes an age-old issue where audio samples could be lost or audio
could temporarily go out of sync in the case of looping videos. When
audio/video data is looping, there's a window between when the audio
data resets its timestamp value and when the video data resets its
timestamp value. This method simply pushes back the audio data while in
that window and does not modify sync, and when it detects that its out
of the loop window it simply forces a resync of the audio data in the
circular buffer.
This ensures that minimal audio data is lost in the loop process, and
minimizes the likelihood of any sort of sync issues associated with
looping.
Instead of applying the resampler offset right away (to each audio
packet), apply the resampler offset when the timestamps are converted to
system timestamps. This fixes an issue where if audio timestamps reset
to 0 (for whatever reason), the offset would cause the timestamp to go
in to the negative.
Mostly only used for transitions with the intention of automatically
creating transitions which don't require configuration, returns whether
the source has any properties or not (whether it's configurable)
(Note: This commit also modifies UI)
Instead of using signals, use designated callback lists for audio
capture and audio control helpers. Signals aren't suitable here due to
the fact that signals aren't meant for things that happen every frame or
things that happen every time audio/video is received. Also prevents
audio from being allocated every time these functions are called due to
the calldata structure.
Transition sources are implemented by registering a source type as
OBS_SOURCE_TYPE_TRANSITION. They're automatically marked as video
composite sources, and video_render/audio_render callbacks must be set
when registering the source. get_width and get_height callbacks are
unused for these types of sources, as transitions automatically handle
width/height behind the scenes with the transition settings.
In the video_render callback, the helper function
obs_transition_video_render is used to assist in automatically
processing and rendering the audio. A render callback is passed to the
function, which in turn passes to/from textures that are automatically
rendered in the back-end.
Similarly, in the audio_render callback, the helper function
obs_transition_audio_render is used to assist in automatically
processing and rendering the audio. Two mix callbacks are used to
handle how the source/destination sources are mixed together. To ensure
the best possible quality, audio processing is per-sample.
Transitions can be set to automatically resize, or they can be set to
have a fixed size. Sources within transitions can be made to scale to
the transition size (with or without aspect ratio), or to not scale
unless they're bigger than the transition. They can have a specific
alignment within the transition, or they just default to top-left.
These features are implemented for the purpose of extending transitions
to also act as "switch" sources later, where you can switch to/from two
different sources using the transition animation.
Planned (but not yet implemented and lower priority) features:
- "Switch" transitions which allow the ability to switch back and forth
between two sources with a transitioning animation without discarding
the references
- Easing options to allow the option to transition with a bezier or
custom curve
- Manual transitioning to allow the front-end/user to manually control
the transition offset
(Note: test and UI are also modified by this commit)
API Changed (removed "enum obs_source_type type" parameter):
-------------------------
obs_source_get_display_name
obs_source_create
obs_get_source_output_flags
obs_get_source_defaults
obs_get_source_properties
Removes the "type" parameter from these functions. The "type" parameter
really doesn't serve much of a purpose being a parameter in any of these
cases, the type is just to indicate what it's used for.
The new audio subsystem fixes two issues:
- First Primary issue it fixes is the ability for parent sources to
intercept the audio of child sources, and do custom processing on
them. The main reason for this was the ability to do custom
cross-fading in transitions, but it's also useful for things such as
side-chain effects, applying audio effects to entire scenes, applying
scene-specific audio filters on sub-sources, and other such
possibilities.
- The secondary issue that needed fixing was audio buffering.
Previously, audio buffering was always a fixed buffer size, so it
would always have exactly a certain number of milliseconds of audio
buffering (and thus output delay). Instead, it now dynamically
increases audio buffering only as necessary, minimizing output delay,
and removing the need for users to have to worry about an audio
buffering setting.
The new design makes it so that audio from the leaves of the scene graph
flow to the root nodes, and can be intercepted by parent sources. Each
audio source handles its own buffering, and each audio tick a specific
number of audio frames are popped from the front of the circular buffer
on each audio source. Composite sources (such as scenes) can access the
audio for child sources and do custom processing or mixing on that
audio. Composite sources use the audio_render callback of sources to do
synchronous or deferred audio processing per audio tick. Things like
scenes now mix audio from their sub-sources.
(Note: This commit breaks libobs compilation. Skip if bisecting)
Adds a "composite" source type which is used for sources that composite
one or more sub-sources. The audio_render callback is called for
composite sources to allow those types of sources to do custom
processing of the audio of its sub-sources.
(Note: This commit breaks libobs compilation. Skip if bisecting)
This variable is somewhat redundant. Volume is already known/accessible
to front-ends.
(Note: This commit breaks libobs compilation. Skip if bisecting)
Removes audio lines and stores the circular buffer for the audio on the
source itself.
(Note: This commit breaks libobs compilation. Skip if bisecting)
The mixers that a source was assigned to were originally stored in the
audio line. This will store it in the sources themselves instead.
Originally this was programmed to call the recursive height/width
functions if the source type was an input with the intention of not
calling it on filters, but instead of doing that just program it to do
just that: only call the recursive height/width functions if it's not a
filter.
This was originally used for calculating audio volume if transitions
were active, but transitions won't work that way so tracking the active
transitions is no longer needed.
Renames:
----------------------------------------
obs_source_add_child
obs_source_remove_child
obs_source_enum_sources
obs_source_enum_tree
obs_source_info::enum_sources
To:
----------------------------------------
obs_source_add_active_child
obs_source_remove_active_child
obs_source_enum_active_sources
obs_source_enum_active_tree
obs_source_info::enum_active_sources
These functions/callbacks had misleading names: they originally implied
any child sources, when they actually meant active child sources that
are being used to render video or audio. It's important that the
function names represent their actual purpose.
(Note: This commit breaks UI compilation. Skip if bisecting)
API Removed:
------------------------
obs_add_source
API Changed:
------------------------
obs_source_remove: Now just marks/signals a source for removal
The concept of "user sources" is flawed: it was something that the
front-end was forced to deal with if it wanted to automate source
saving/loading, and often it had to code around it. That's not how
saving/loading should work, a front-end should be allowed to manage
lists of sources in the way it explicitly chooses, and it should be able
to choose which sources it wants to save/load.
This shouldn't happen anymore because crop was fixed, but if a filter
returns 0x0 size and is invalid it shouldn't stop the filter chain.
Instead, it should just be skipped.
When an async video source is about to be rendered, the async texture
should be updated before any effect filtering occurs, rather than right
when it's about to render.
Fixes a few bugs:
- If the async texture hadn't drawn for its first time, and the source
has an effect filter, it would never end up rendering the first
frame due to the fact that it would fail on obs-source.c:2434 for the
first filter, causing it to never actually render the source, and thus
never get to a point in which it could call set_async_texture_size to
establish the async texture width/height for the first time.
- Any time the async texture size changed, it would only update the
async texture size at the end of the filter loop, which means that the
first frame after a size change would use the old size for the filters
rather than update to the new size right away.
To prevent confusion with the new obs_source_valid function which
displays a warning, rename the "source_valid" function to "data_valid"
to emphasize that it's checking for the validity of the internal data as
well as the source itself.
API removed:
--------------------
gs_effect_t *obs_get_default_effect(void);
gs_effect_t *obs_get_default_rect_effect(void);
gs_effect_t *obs_get_opaque_effect(void);
gs_effect_t *obs_get_solid_effect(void);
gs_effect_t *obs_get_bicubic_effect(void);
gs_effect_t *obs_get_lanczos_effect(void);
gs_effect_t *obs_get_bilinear_lowres_effect(void);
API added:
--------------------
gs_effect_t *obs_get_base_effect(enum obs_base_effect effect);
Summary:
--------------------
Combines multiple near-identical functions into a single function with
an enum parameter.
API changed from:
obs_source_info::get_name(void)
obs_output_info::get_name(void)
obs_encoder_info::get_name(void)
obs_service_info::get_name(void)
API changed to:
obs_source_info::get_name(void *type_data)
obs_output_info::get_name(void *type_data)
obs_encoder_info::get_name(void *type_data)
obs_service_info::get_name(void *type_data)
This allows the type data to be used when getting the name of the
object (useful for plugin wrappers primarily).
NOTE: Though a parameter was added, this is backward-compatible with
older plugins due to calling convention. The new parameter will simply
be ignored by older plugins, and the stack (if used) will be cleaned up
by the caller.
When an audio filter is applied to a video source that also has
accompanying audio, it would cause the video from the source to stop
rendering.
The original code this was from was to prevent audio-only sources from
rendering video, but I neglected to make sure that this would not apply
to filters, and thus when an audio filter is on a source with video, the
code would kill the video.
Due to the fact that async timestamps themselves can be susceptible to
minor jitter from certain types of inputs, increase the allowable jitter
compensation value to ensure that the rendered frame timing from async
video sources is always as close as possible to the compositor.
When the framerate of the source is the same as the framerate as the
compositor, this (combined with the fact that clamped video timing now
being used with async video frames) helps ensure that buffered async
video sources will sync up their rendering to the compositor as
accurately as possible despite jitter from the source's timestamps.
If there is no jitter in the source's timestamps then it'll always sync
up perfectly with the compositor, thanks to clamped video timing.
When playing back buffered async frames, this reduces the probability
that new frames will be missed/skipped due to jitter in the system
timestamps.
If a buffered async source is playing at the same framerate as the
compositor and there is no jitter in the async source's timestamps, then
the async source will play back perfectly in sync with the compositor
thanks to this change, ensuring that there's no skipped or missed frames
in video playback.
When buffering is enabled for an async video source, sometimes minor
drift in timestamps or unexpected delays to frames can cause frames to
slowly buffer more and more in memory, in some cases eventually causing
the system to run out of memory.
The circumstances in which this can happen seems to depend on both the
computer and the devices in use. So far, the only known circumstances
in which this happens are with heavily buffered devices, such as
hauppauge, where decoding can sometimes take too long and cause
continual frame playback delay, and thus continual buffering until
memory runs out. I've never been able to replicate it on any of my
machines however, even after hours of testing.
This patch is a precautionary measure that puts a hard limit on the
number of async frames that can be currently queued to prevent any case
where memory might continually build for whatever reason. If it goes
over the limit, it clears the cache to reset the buffering.
I had a user with this problem test this patch with success and positive
feedback, and the intervals between buffering resets were long to where
it wasn't even noticeable while streaming/recording.
Ideally when decoding frames (such as from those devices), frame
dropping should be used to ensure playback doesn't incur extra delay,
although this sort of hard limit on the frame cache should still be
implemented regardless just as a safety precaution. For DirectShow
encoded devices I should just switch to faruton's libff for decoding and
enable the frame dropping options. It would probably explain why no
one's ever reported it for the media source, and pretty much only from
DirectShow device usage.
Fixes a crash that could happen if any of the mutexes are used in the
create callback, or before the obs_source_init function is called.
I'm not sure how this function order slipped because it seems fairly
obvious that these mutexes should be created before the create callback.
Had this crash happen to me when creating a WASAPI output source, the
create callback of the WASAPI source creates a thread which outputs
audio, and that thread managed to call obs_source_output_audio before
the obs_source_init function was called, which in turn caused it to try
to use a null mutex.
When caching a new frame, keep a reference to the frame while copying to
ensure that the frame is not potentially destroyed for whatever reason
while that data is being copied.
The obs_source::async_reset_texture variable can cause a data race
between threads to occur because it could be set to true in one thread
then changed back to false in another thread. This could cause the
async texture to not update its size when it's supposed to, which can
cause a crash or corruption when copying data from a frame of a
differing size.
The solution to this is to:
- Delete the async_reset_texture variable, and make the
set_async_texture_size function change the texture size if the
async_width, async_height, or async_format variables differ from the
frame's width/height/format. Those variables are then only ever set
in the libobs graphics thread.
- Make the cache_video function use separate variables from other
functions to detect a change in size (due to the fact that the texture
size should only be resized in the libobs graphics thread). These
variables are async_cache_width, async_cache_height, and
async_cache_format, which are only be set in the thread that calls
obs_source_output_video.
How to replicate the data race:
- On OSX, use window capture on a textedit window, then continually
resize the textedit window.
Due to a bad 'if' expression, when a filter that is not last in the
chain is disabled or being bypassed, it ends up still calling the
filter's video processing function unintentionally.
This fix makes sure that it only calls the appropriate render functions
if the next filter target is the source, otherwise it will just call
obs_source_video_render to process the next filter in the chain.
How to replicate the bug:
1. Create two crop filters on the same source
2. Give each crop filter a different distinct value
3. Disable both crop filters
4. The image would still be cropped
This fixes an issue where cache frames would not free at all after
having been allocated with no upper limit on the cached frame size. If
cached frames go unused for a specific period of time, they are
deallocated and removed from the cache.
This is preferable to having an upper cache limit due to the potential
for async delay filtering.
Under certain circumstances the cache could be prone to growing too
large unintentionally. Setting a hard maximum limit should prevent
memory from growing if we suddenly get a lot of frames.
Async frames are only swapping when rendering, or when not visible.
This is a flawed design due to the fact that there are certain
circumstances where the source is neither visible nor currently
rendering.
This is what caused a memory leak when scene items were marked as
invisible, because if a source has an async child source and decides not
to render that source for whatever reason, the child source would not
process the async frames at all, and the cache would just grow.
To fix this, simply moving the async frame cycle to tick fixes the issue
due to the fact that tick is always called regardless of circumstance.
Filtering the video before it's output to the texture means that it
happens after all the processing on the timestamps and such of the video
data. This way, the video filter does not have to worry about what's
currently buffered, and it won't affect timing.
When OBS is shutting down, if for some reason the filter is destroyed
before the parent source is destroyed, it would try to remove itself
from the source, but it would decrement the reference and try to destroy
itself again while already in the process of destroying itself.
So, the solution was simply to make sure that if it's removing itself
from the source that it doesn't decrement its own reference.
The "last" filter that's rendered is technically at filter index 0, so
enumeration needs to be from the last index in the list to the first
index in the list.
When rendering a filter to a texture, the target is empty and unused, so
there's no reason for blending to be on when rendering the filter to a
render target.
obs_source_process_filter tried to do everything in a single function,
but the problem is that effect parameters would not properly be
accounted for due to the way it internally draws, therefore it was
necessary to split the functions in to two, you first call
obs_source_process_filter_begin, then you set your effect parameters,
then you finally call obs_source_process_filter_end. This ensures that
when the filter is drawn, that the effect parameters are set.
When the filter chain finally reaches the source and the last filter in
the chain is set to not render directly (meaning it has to render to
texture), it would not render the source with any effect due to the fact
that it expects a filter to be present.
Adds a source callback function that is used when a filter is removed
from a source. This ensures that any data that might be associated with
that source can be cleaned up if necessary.
There are a few more conditions which need to be checked to ensure
whether we can actually bypass or not; in this particular case we are
using the YUV texture shaders, plus the image can also be flipped, so we
can't really use the bypass optimization in those situations.
These functions are primarily for use with filters, filters need to be
able to get the width/height of a target source without it necessarily
getting the post-filtered dimensions.
Previously I had it set so that it would set the async_width and
async_height variables to 0,0 if a null frame occurs, but the problem
with this is that it was inadvertently cause the frame cache to clear
when it starts receiving textures again because it detects it as a size
change.
So instead of changing async_width and async_height to 0 when a null
frame occurs, change it so that if the source is set to an inactive
state, make obs_source_get_width/_get_height return 0 for their
dimensions instead.
When a frame is processed by a filter, it comes directly from the
source's video frame cache. However, if a filter is using or processing
those frames for whatever reason, there would be no guarantee that the
frames would persist during processing, and frames could eventually be
deallocated unexpected, for example when the resolution or format
changes.
So the solution is to implement simple reference counting for the frames
so that the frames will exist until they have been released by any
source or filter that's using them.
Fix a bug where when a source first starts up its async cache, it
unintentionally resets its cache, which means that the first few frames
would be lost.
The wrong function was being used to recurse through the filter chain in
obs_source_process_filter, obs_source_get_[width/height] would get the
post-effect dimensions rather than the pre-effect dimensions.
If the audio data had the same format/samplerate as the obs audio
subsystem, it would fail to simply destroy the resampler and set it to
NULL, and then any audio data going through would use the resampler that
was being used before that, causing audio to become garbage.
This bug only started appearing when I recently changed the libobs
internal audio subsystem format to non-interleaved floating point, which
is a common format, and thus caused this bug to actually occur more
often.
I when a source has both async audio/video capability, it would ignore
audio until the video has started. There's really no need to do this,
when the video starts it'll just fix up the timing automatically.
This should fix the case where things like the media source would not be
able to play audio-only files.
Core API functions changed:
-----------------------------
EXPORT bool obs_reset_audio(struct audio_output_info *aoi);
EXPORT bool obs_get_audio_info(struct audio_output_info *aoi);
To:
-----------------------------
EXPORT bool obs_reset_audio(const struct obs_audio_info *oai);
EXPORT bool obs_get_audio_info(struct obs_audio_info *oai);
Core structure added:
-----------------------------
struct obs_audio_info {
uint32_t samples_per_sec;
enum speaker_layout speakers;
uint64_t buffer_ms;
};
Non-interleaved (planar) floating point output is standard with audio
filtering, so to prevent audio filters from having to worry about
different audio format implementations and for the sake consistency
between user interfaces, make it so that audio is always set to
non-interleaved floating point output.
This code was originally meant to skip some checks as an optimization,
but it did not account for async video sources and would call
obs_source_default_render (which is only for synchronous video sources),
this fixes the issue by just calling obs_source_video_render.
The obs_source_get_width and obs_source_get_height functions need to
account for filters that may change their width/height such as a crop
filter or something similar.
As a side effect of this commit, because these functions need to lock
the filter mutex, these functions can no longer be used with a
const-qualified obs_source_t pointer.
This function is somewhat big and is a function called conditionally,
therefore having it be inline is somewhat of a waste if it's not always
being called.
Graphics data has to be freed inside of an active graphics context,
otherwise the resources will not be freed; in the future, we should
probably make sure that the graphics subsystem automatically
asserts/warns about this scenario.
I was iterating through the obs_source::filters array to remove filters,
but every time I removed a filter, it would remove itself from the
array, therefore it would end up skipping items in the array, therefore
leaving filters unreleased. This just removes the first filter until
all filters have been removed.
For the show/hide and activate/deactivate callbacks, schedule these
callbacks to only be called from within the video thread rather than in
a separate thread. This ensures that any potential graphics activity
that occurs within them is kept in the same thread.
obs_source_inc_showing and obs_source_dec_showing are used to indicate
that a source is showing or no longer being shown in a particular area
of the program outside from the main render view.
One could use an obs_view, but I felt like it was unnecessary because
using an obs_view just to display a single source feels somewhat
superfluous.
API changed:
--------------------------
void obs_output_set_audio_encoder(
obs_output_t *output,
obs_encoder_t *encoder);
obs_encoder_t *obs_output_get_audio_encoder(
const obs_output_t *output);
obs_encoder_t *obs_audio_encoder_create(
const char *id,
const char *name,
obs_data_t *settings);
Changed to:
--------------------------
/* 'idx' specifies the track index of the output */
void obs_output_set_audio_encoder(
obs_output_t *output,
obs_encoder_t *encoder,
size_t idx);
/* 'idx' specifies the track index of the output */
obs_encoder_t *obs_output_get_audio_encoder(
const obs_output_t *output,
size_t idx);
/* 'mixer_idx' specifies the mixer index to capture audio from */
obs_encoder_t *obs_audio_encoder_create(
const char *id,
const char *name,
obs_data_t *settings,
size_t mixer_idx);
Overview
--------------------------
This feature allows multiple audio mixers to be used at a time. This
capability was able to be added with surprisingly very little extra
overhead. Audio will not be mixed unless it's assigned to a specific
mixer, and mixers will not mix unless they have an active mix
connection.
Mostly this will be useful for being able to separate out specific audio
for recording versus streaming, but will also be useful for certain
streaming services that support multiple audio streams via RTMP.
I didn't want to use a variable amount of mixers due to the desire to
reduce heap allocations, so currently I set the limit to 4 simultaneous
mixers; this number can be increased later if needed, but honestly I
feel like it's just the right number to use.
Sources:
Sources can now specify which audio mixers their audio is mixed to; this
can be a single mixer or multiple mixers at a time. The
obs_source_set_audio_mixers function sets the audio mixer which an audio
source applies to. For example, 0xF would mean that the source applies
to all four mixers.
Audio Encoders:
Audio encoders now must specify which specific audio mixer they use when
they encode audio data.
Outputs:
Outputs that use encoders can now support multiple audio tracks at once
if they have the OBS_OUTPUT_MULTI_TRACK capability flag set. This is
mostly only useful for certain types of RTMP transmissions, though may
be useful for file formats that support multiple audio tracks as well
later on.
The temporary unoptimized code we were using before just completely
allocated a new copy of each frame every single time a new async frame
was output by the source plugin. This just creates a cache of frames as
needed for the current format/width/height to minimize the allocation
and deallocation. If new frames come in that are of a different
format/width/height, it'll just clear the cache. This is a fairly
important optimization.
all the async video related stuff usually started with async_*, and
there were two that didn't. So I just renamed them so they have the
same naming convention
If an async video source stops video for whatever reason, it would get
stuck on the last frame that was played. This was particularly awkward
when I wanted to give the user the ability to deactivate a source such
as a webcam because it would get stuck on the last frame.
I unintentionally made it use obs_source::sample_info instead of using
the actual target channel count, which is designated by the OBS output
sampler info. obs_source::sample_info is actually used to indicate the
currently set sampler information for incoming samples. So if a source
is outputting 5.1 channel 48khz audio, and OBS is running at stereo
44.1khz, then the obs_source::sample_info value would be set to
5.1/48khz, not the other way around. It indicates what the source
itself is running at, not what OBS is running at.
I suppose the variable needs a better name because even I used it
incorrectly despite actually having been the one who wrote it.
The copy_audio_data function really shouldn't be inlined because it's
being called twice. It's somewhat unnecessary, I think I left it inline
by accident.
This changes the way source volume handles transitioning between being
active and inactive states.
The previous way that transitioning handled volume was that it set the
presentation volume of the source and all of its sub-sources to 0.0 if
the source was inactive, and 1.0 if active. Transition sources would
then also set the presentation volume for sub-sources to whatever their
transitioning volume was. However, the problem with this is that the
design didn't take in to account if the source or its sub-sources were
active anywhere else, so because of that it would break if that ever
happened, and I didn't realize that when I was designing it.
So instead, this completely overhauls the design of handling
transitioning volume. Each frame, it'll go through all sources and
check whether they're active or inactive and set the base volume
accordingly. If transitions are currently active, it will actually walk
the active source tree and check whether the source is in a
transitioning state somewhere.
- If the source is a sub-source of a transition, and it's not active
outside of the transition, then the transition will control the
volume of the source.
- If the source is a sub-source of a transition, but it's also active
outside of the transition, it'll defer to whichever is louder.
This also adds a new callback to the obs_source_info structure for
transition sources, get_transition_volume, which is called to get the
transitioning volume of a sub-source.
The reason to keep a reference counter for transitions is due to an
optimization I'm planning on when calculating transition volumes. I'm
planning on walking the source tree to be able to calculate the current
base volume of a source, but *only* if there are transitions active,
because the only time that the volume can be anything other than 1.0
or 0.0 is when there are active transitions, which may change the base
volume of a source.
When the presentation volume is set for a source, it's set for all of
its children and their children. The original intention for doing this
was to be able to use it for transitioning, but honestly it's just bad
design, and I feel there are better ways to handle transitioning volume.
Changed the design from using obs_source::enum_refs to just simply
preventing infinite source recursion in general, rather than allowing it
through the enum_refs variable. obs_source_add_child has been changed
so that it now returns a boolean, and if the function fails, it means
that the child cannot be added due to that potential recursion.
Remove the calculation of volume levels and the corresponding signal
from obs_source since this is now handled in the volume meter.
Code that is interested in the volume levels can either use the
volmeter provided from obs_audio_controls or use the audio_data signal
to gain access to the raw audio data.
This adds a new signal to (audio) sources which is emitted whenever new
audio data is received from the source. This enables other code that is
interested in the raw audio data to directly access it when it becomes
available.
This replaces the old code for the audio meter that was using
calculations in two different places with the new audio meter api.
The source signal will now emit simple levels instead of dB values,
in order to avoid dB conversion and calculation in the source.
The GUI on the other hand now expects simple position values from
the volume meter api with no knowledge about dB calculus either.
That way all code that handles those conversions is in one place,
with the option to easily add new mappings that can be used
everywhere.
Because a vec3 structure can contain a __m128 variable and not the
expected three floats x, y, and z, you must use vec3_set when
setting a value for a vec3 structure to ensure that it uses the proper
intrinsics internally if necessary.
If you look at the previous commits, you'll see I had added
obs_source_draw before. For custom drawn sources in particular, each
time obs_source_draw was called, it would restart the effect and its
passes for each draw call, which was not optimal. It should really use
the effect functions for that. I'll have to add a function to simplify
effect usage.
I also realized that including the color matrix parameters in
obs_source_draw made the function kind of messy to use; instead,
separating the color matrix stuff out to
obs_source_draw_set_color_matrix feels a lot more clean.
On top of that, having the ability to set the position would be nice to
have as well, rather than having to mess with the matrix stuff each
time, so I also added that for the sake of convenience.
obs_source_draw will draw a texture sprite, optionally of a specific
size and/or at a specific position, as well as optionally inverted. The
texture used will be set to the 'image' parameter of whatever effect is
currently active.
obs_source_draw_set_color_matrix will set the color matrix value if the
drawing requires color matrices. It will set the 'color_matrix',
'color_range_min', and 'color_range_max' parameters of whatever effect
is currently active.
Overall, these feel much more clean to use than the previous iteration.
This function simplifies drawing textures for sources in order to help
reduce boilerplate code. If a source is a custom drawn source, it will
automatically set up the effect to draw the sprite. If it's not a
custom drawn source, it will simply draw the sprite as per normal. If
the source uses a specific color matrix, it will also handle that as
well.
This moves the 'flags' variable from the obs_source_frame structure to
the obs_source structure, and allows user flags to be set for a specific
source. Having it set on the obs_source_frame structure didn't make
much sense.
OBS_SOURCE_UNBUFFERED makes it so that the source does not buffer its
async video output in order to try to play it on time. In other words,
frames are played as soon as possible after being received.
Useful when you want a source to play back as quickly as possible
(webcams, certain types of capture devices)
This reverts commit c3f4b0f018.
The obs_source_frame should not need to take flags to do this. This
shouldn't be a setting associated with the frame, but rather a setting
associated with the source itself. This was the wrong approach to
solving this particular problem.
If audio timestamps are within the operating system timing threshold,
always use those values directly as a timestamp, and do not apply the
regular jump checks and timing adjustments that normally occur.
This potentially fixes an issue with plugins that use OS timestamps
directly as timestamp values for their audio samples, and bypasses the
timing conversions to system time for the audio line and uses it
directly as the timestamp value. It prevents those calculations from
potentially affecting the audio timestamp value when OS timestamps are
used.
For example, if the first set of audio samples from the audio source
came in delayed, while the subsequent samples were not delayed, those
first samples could have potentially inadvertently triggered the timing
adjustments, which would affect all subsequent audio samples.
This combines the 'direct' timestamp variance threshold with the maximum
timestamp jump threshold (or rather just removes the max timestamp jump
threshold and uses the timestamp variance threshold for both timestamp
jumps and detecting timestamps).
The reason why this was done was because a timestamp jump could occur at
a higher threshold than the threshold used for detecting OS timestamps
within a certain threshold. If timestamps got between those two
thresholds it kind of became a weird situation where timestamps could be
sort of 'stuck' back or forward in time more than intended. Better to
be consistent and use the same threshold for both values.
Add 'flags' member variable to obs_source_frame structure.
The OBS_VIDEO_UNBUFFERED flags causes the video to play back as soon as
it's received (in the next frame playback), causing it to disregard the
timestamp value for the sake of video playback (however, note that the
video timestamp is still used for audio synchronization if audio is
present on the source as well).
This is partly a convenience feature, and partly a necessity for certain
plugins (such as the linux v4l plugin) where timestamp information for
the video frames can sometimes be unreliable.
Due to a small error in the timestamp smoothing code the timestamp of
audio packages that were too early was always set to the next expected
timestamp, even if the difference was bigger than the smoothing threshold.
This would cause obs to simply append all audio data to the buffer even if
the real timestamp was way smaller than the next that was expected.
This should reduce corruption problems with for example the pulseaudio
plugin, which resends data under certain conditions.
obs_source_update_properties should be called by sources when property
values change, e.g. a capture device source would use this when it
detects a new capture device (in case its properties contain a list of
available capture devices or similar)
This Fixes a minor flaw with the API where data had to always be mutable
to be usable by the API.
Functions that do not modify the fundamental underlying data of a
structure should be marked as constant, both for safety and to signify
that the parameter is input only and will not be modified by the
function using it.
Typedef pointers are unsafe. If you do:
typedef struct bla *bla_t;
then you cannot use it as a constant, such as: const bla_t, because
that constant will be to the pointer itself rather than to the
underlying data. I admit this was a fundamental mistake that must
be corrected.
All typedefs that were pointer types will now have their pointers
removed from the type itself, and the pointers will be used when they
are actually used as variables/parameters/returns instead.
This does not break ABI though, which is pretty nice.
This helps ensure that an asynchronous video source is played as close
to its framerate as possible, reduces the risk of duplication as
much as possible, and helps to ensure that playback is as smooth as
possible.
This prevents multiple needless calls to obs_source_get_frame and other
functions. If the texture has already been processed, then just render
it as-is in any subsequent calls to obs_source_video_render.
This is actually unnecessary now that there's a hard limit on the
maximum offset in which audio can be inserted.
This also assumes too much about the audio; it assumes audio is always
on, where as with some devices (such as the elgato) audio is not on
until the stream starts, and when the video has already incremented the
counter.
If the audio didn't start at the 0 timestamp, it would misinterpret it
as a timestamp jump because obs_source::next_audio_ts_min is set to 0 on
creation. Timestamp starting values should be allowed to start at any
arbitrary value.
Instead of having functions like obs_signal_handler() that can fail to
properly specify their actual intent in the name (does it signal a
handler, or does it return a signal handler?), always prefix functions
that are meant to get information with 'get' to make its functionality
more explicit.
Previous names: New names:
-----------------------------------------------------------
obs_audio obs_get_audio
obs_video obs_get_video
obs_signalhandler obs_get_signal_handler
obs_prochandler obs_get_proc_handler
obs_source_signalhandler obs_source_get_signal_handler
obs_source_prochandler obs_source_get_proc_handler
obs_output_signalhandler obs_output_get_signal_handler
obs_output_prochandler obs_output_get_proc_handler
obs_service_signalhandler obs_service_get_signal_handler
obs_service_prochandler obs_service_get_proc_handler
Changed:
- obs_source_gettype
To:
- enum obs_source_type obs_source_get_type(obs_source_t source);
- const char *obs_source_get_id(obs_source_t source);
This function was inconsistent for a number of reasons. First, it
returns both the ID and the type of source (input/transition/filter),
which is inconsistent with the name of "get type". Secondly, the
'squishy' naming convention which has just turned out to be bad
practice and causes inconsistencies. So it's now replaced with two
functions that just return the type and the ID.
Prefix with obs_ for the sake of consistency
Renamed enums:
- order_movement (now obs_order_movement)
Affected functions:
- obs_source_filter_setorder
- obs_sceneitem_setorder
Renamed functions:
- obs_source_getframe (rename to obs_source_get_frame)
- obs_source_releaseframe (rename to obs_source_release_frame)
For the sake of consistency and helping to get rid of the "squishy
function name" issue
With the recent change to module handling by BtbN, I felt that having
this information might be useful in case someone is actually using make
install to set up their libraries.
NOTE: In texture_setimage, I had to move variables to the top of the
scope because microsoft's C compiler will give the legacy C90 error of:
'illegal use of this type as an expression'.
To sum it up, microsoft's C compiler is still utter garbage.
Similar to the shader functions, the effect parameter functions take
the effect as a parameter. However, the effect parameter is pretty
pointless, because the effect parameter.. parameter stores the effect
pointer interally.
The locale parameter was a mistake, because it puts extra needless
burden upon the module developer to have to handle this variable for
each and every single callback function. The parameter is being removed
in favor of a single centralized module callback function that
specifically updates locale information for a module only when needed.
This replaces the older code which simply queried the max volume level
value for any given audio.
I'm still not 100% sure on if this is how I want to approach the
problem, particularly, whether this should be done in obs_source or in
audio_line, but it can always be moved later if needed.
This uses the calculations by the awesome Bill Hamilton that OBS1 used
for its volume levels. It calculates the current max (level),
magnitude, and current peak. This data then can be used to create
awesome volume meter controls later on.
NOTE: Will probably need optimization, does one float at a time right
now.
Also, change some of the naming conventions. I actually need to change
a lot of the naming conventions in general so that all words are
separated by underscores. Kind of a bad practice there on my part.
When a source's private data is being created by a module, it wasn't
able to call most source functions because most functions rely on the
obs_source_info part of the context to be set. This fixes that issue.
It was strange that this wasn't already the case because the other
context types already did the same thing.
This uses the reverse planar YUV 4:2:0 conversion shader to output a YUV
texture without having to convert it via CPU. Again, this will reduce
video upload bandwidth usage to 37.5% of the original rate. I suspect
this will be particularly useful for when an FFmpeg or libav input
plugin for playing videos is made.
NOTE: There's an issue with certain texture sizes right now I haven't
been able to identify, if the full size of texture data divided by the
base texture width is an uneven number, the V chroma plane seems like it
can potentially shift, though I only had this happen with 160x90
resolution C920. Almost all resolutions tend to be even. Needs further
testing with more devices that support planar YUV 4:2:0 output.
First, if the private data of the source fails to be created, then do
not destroy the source. If the source is destroyed, all the user's data
associated with that source is lost, which could end up being a
potential problem. Instead, let it linger as a 'dead' source until the
user chooses to fix the problem (though this should never really happen,
the source module functions should be programmed to handle this
scenario)
Secondly, rename new_frame_ready to ready_async_frame, and fix a
potential memory leak with it.
obs_source_output_video can cause cached frames to be freed twice if
called with a partially destroyed source, among other undesirable
effects; freeing the source private data right after the destroy signal
has been processed ensures proper behavior
- Add volume control
These volume controls are basically nothing more than sliders. They
look terrible and hopefully will be as temporary as they are
terrible.
- Allow saving of specific non-user sources via obs_load_source and
obs_save_source functions.
- Save data of desktop/mic audio sources (sync data, volume data, etc),
and load the data on startup.
- Make it so that a scene is created by default if first time using the
application. On certain operating systems where supported, a default
capture will be created. Desktop capture on mac, particularly. Not
sure what to do about windows because monitor capture on windows 7 is
completely terrible and is bad to start users off with.
If a source with async video wasn't currently active, it would endlessly
buffer the video data, which would cause memory to grow endlessly until
available memory was extinguished.
This really needs to be replaced with a proper caching mechanism at some
point.
This saves scenes/sources from json on exit, and properly loads it back
up when starting up the program again, as well as the currently active
scene.
I had to add a 'load' and 'save' callback to the source interface
structure because I realizes that certain sources (such as scenes)
operate different with their saved data; scenes for example would have
to keep track of their settings information constantly, and that was
somewhat unacceptable to make it functional.
The optional 'load' callback will be called only after having loaded
setttings specifically from file/imported data, and the 'save' function
will be called only specifically when data actually needs to be saved.
I also had to adjust the obs_scene code so that it's a regular input
source type now, and I also modified it so that it doesn't have some
strange custom creation code anymore. The obs_scene_create function is
now simply just a wrapper for obs_source_create. You could even create
a scene with obs_source_create manually as well.
Add API for streaming services. The services API simplifies the
creation of custom service features and user interface.
Custom streaming services later on will be able to do things such as:
- Be able to use service-specific APIs via modules, allowing a more
direct means of communicating with the service and requesting or
setting service-specific information
- Get URL/stream key via other means of authentication such as OAuth,
or be able to build custom URLs for services that require that sort
of thing.
- Query information (such as viewer count, chat, follower
notifications, and other information)
- Set channel information (such as current game, current channel title,
activating commercials)
Also, I reduce some repeated code that was used for all libobs objects.
This includes the name of the object, the private data, settings, as
well as the signal and procedure handlers.
I also switched to using linked lists for the global object lists,
rather than using an array of pointers (you could say it was..
pointless.) ..Anyway, the linked list info is also stored in the shared
context data structure.
Before, async video sources would flicker because they were only being
drawn when they were updated. So when updated, they'd draw that frame,
then it would stop drawing it until it updated again. This fixes that
issue and they should now draw properly.
Also, fix a few other minor bugs and issues relating to async video,
and make it so that non-async video filters can be properly applied to
them.
For the purposes of testing, change the 'test-random' source to an async
video source that updates every quarter of a second with a new random
face.
Also fix a bug where non-async video sources wouldn't have filter
effects applied properly.
- Implement the RTMP output module. This time around, we just use a
simple FLV muxer, then just write to the stream with RTMP_Write.
Easy and effective.
- Fix the FLV muxer, the muxer now outputs proper FLV packets.
- Output API:
* When using encoders, automatically interleave encoded packets
before sending it to the output.
* Pair encoders and have them automatically wait for the other to
start to ensure sync.
* Change 'obs_output_signal_start_fail' to 'obs_output_signal_stop'
because it was a bit confusing, and doing this makes a lot more
sense for outputs that need to stop suddenly (disconnections/etc).
- Encoder API:
* Remove some unnecessary encoder functions from the actual API and
make them internal. Most of the encoder functions are handled
automatically by outputs anyway, so there's no real need to expose
them and end up inadvertently confusing plugin writers.
* Have audio encoders wait for the video encoder to get a frame, then
start at the exact data point that the first video frame starts to
ensure the most accrate sync of video/audio possible.
* Add a required 'frame_size' callback for audio encoders that
returns the expected number of frames desired to encode with. This
way, the libobs encoder API can handle the circular buffering
internally automatically for the encoder modules, so encoder
writers don't have to do it themselves.
- Fix a few bugs in the serializer interface. It was passing the wrong
variable for the data in a few cases.
- If a source has video, make obs_source_update defer the actual update
callback until the tick function is called to prevent threading
issues.
Improve the properties API so that it can actually respond somewhat to
user input. Maybe later this might be further improved or replaced with
something script-based.
When creating a property, you can now add a callback to that property
that notifies when the property has been changed in the user interface.
Return true if you want the properties to be refreshed, or false if not.
Though now that I think about it I doubt there would ever be a case
where you would have this callback and *not* refresh the properties.
Regardless, this allows functions to change the values of properties or
settings, or enable/disable/hide other property controls from view
dynamically.
