Frame-threaded decoders with inter-frame dependencies
use the ThreadFrame API for syncing. It works as follows:
During init each thread allocates an AVFrame for every
ThreadFrame.
Thread A reads the header of its packet and allocates
a buffer for an AVFrame with ff_thread_get_ext_buffer()
(which also allocates a small structure that is shared
with other references to this frame) and sets its fields,
including side data. Then said thread calls ff_thread_finish_setup().
From that moment onward it is not allowed to change any
of the AVFrame fields at all any more, but it may change
fields which are an indirection away, like the content of
AVFrame.data or already existing side data.
After thread A has called ff_thread_finish_setup(),
another thread (the user one) calls the codec's update_thread_context
callback which in turn calls ff_thread_ref_frame() which
calls av_frame_ref() which reads every field of A's
AVFrame; hence the above restriction on modifications
of the AVFrame (as any modification of the AVFrame by A after
ff_thread_finish_setup() would be a data race). Of course,
this av_frame_ref() also incurs allocations and therefore
needs to be checked. ff_thread_ref_frame() also references
the small structure used for communicating progress.
This av_frame_ref() makes it awkward to propagate values that
only become known during decoding to later threads (in case of
frame reordering or other mechanisms of delayed output (like
show-existing-frames) it's not the decoding thread, but a later
thread that returns the AVFrame). E.g. for VP9 when exporting video
encoding parameters as side data the number of blocks only
becomes known during decoding, so one can't allocate the side data
before ff_thread_finish_setup(). It is currently being done afterwards
and this leads to a data race in the vp9-encparams test when using
frame-threading. Returning decode_error_flags is also complicated
by this.
To perform this exchange a buffer shared between the references
is needed (notice that simply giving the later threads a pointer
to the original AVFrame does not work, because said AVFrame will
be reused lateron when thread A decodes the next packet given to it).
One could extend the buffer already used for progress for this
or use a new one (requiring yet another allocation), yet both
of these approaches have the drawback of being unnatural, ugly
and requiring quite a lot of ad-hoc code. E.g. in case of the VP9
side data mentioned above one could not simply use the helper
that allocates and adds the side data to an AVFrame in one go.
The ProgressFrame API meanwhile offers a different solution to all
of this. It is based around the idea that the most natural
shared object for sharing information about an AVFrame between
decoding threads is the AVFrame itself. To actually implement this
the AVFrame needs to be reference counted. This is achieved by
putting a (ownership) pointer into a shared (and opaque) structure
that is managed by the RefStruct API and which also contains
the stuff necessary for progress reporting.
The users get a pointer to this AVFrame with the understanding
that the owner may set all the fields until it has indicated
that it has finished decoding this AVFrame; then the users are
allowed to read everything. Every decoder may of course employ
a different contract than the one outlined above.
Given that there is no underlying av_frame_ref(), creating
references to a ProgressFrame can't fail. Only
ff_thread_progress_get_buffer() can fail, but given that
it will replace calls to ff_thread_get_ext_buffer() it is
at places where errors are already expected and properly
taken care of.
The ProgressFrames are empty (i.e. the AVFrame pointer is NULL
and the AVFrames are not allocated during init at all)
while not being in use; ff_thread_progress_get_buffer() both
sets up the actual ProgressFrame and already calls
ff_thread_get_buffer(). So instead of checking for
ThreadFrame.f->data[0] or ThreadFrame.f->buf[0] being NULL
for "this reference frame is non-existing" one should check for
ProgressFrame.f.
This also implies that one can only set AVFrame properties
after having allocated the buffer. This restriction is not deep:
if it becomes onerous for any codec, ff_thread_progress_get_buffer()
can be broken up. The user would then have to get a buffer
himself.
In order to avoid unnecessary allocations, the shared structure
is pooled, so that both the structure as well as the AVFrame
itself are reused. This means that there won't be lots of
unnecessary allocations in case of non-frame-threaded decoding.
It might even turn out to have fewer than the current code
(the current code allocates AVFrames for every DPB slot, but
these are often excessively large and not completely used;
the new code allocates them on demand). Pooling relies on the
reset function of the RefStruct pool API, it would be impossible
to implement with the AVBufferPool API.
Finally, ProgressFrames have no notion of owner; they are built
on top of the ThreadProgress API which also lacks such a concept.
Instead every ThreadProgress and every ProgressFrame contains
its own mutex and condition variable, making it completely independent
of pthread_frame.c. Just like the ThreadFrame API it is simply
presumed that only the actual owner/producer of a frame reports
progress on said frame.
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@outlook.com>
For hardware cases where we are forced to have a fixed pool of frames
allocated up-front (such as array textures on decoder output), suggest
a possible workaround to the user if an allocation fails because the
pool is exhausted.
Avoids allocations and frees and error checks for said allocations;
also avoids a few indirections and casts.
Reviewed-by: Anton Khirnov <anton@khirnov.net>
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@outlook.com>
This decoding flag makes decoders drop all frames after a parameter
change, but what exactly constitutes a parameter change is not well
defined and will typically depend on the exact use case.
This functionality then does not belong in libavcodec, but rather in
user code
Decoders will currently warn if an audio decoder not marked with
AV_CODEC_CAP_SUBFRAMES consumes less than the whole packet, but
* this happens for regular files
* this has no negative consequences
* there is no meeaningful action that can or should be taken in response
The warning is thus useless noise.
The idea behind last_pkt_props was to store the properties of the last packet
fed to the decoder. Any sort of queueing required by CODEC_CAP_DELAY decoders
that consume several packets before they start outputting frames should be done
by the decoders in question. An example of this is libdav1d.
This is required for the following commits that will fix last_pkt_props in
frame threading scenarios, as well as maintain its contents during flush.
This revers commit 022a12b306.
Signed-off-by: James Almer <jamrial@gmail.com>
Some audio codecs work with atomic units that decode to a fixed
number of audio samples with this number being so small that it is
common to put multiple of these atoms into one packet. In these
cases it makes no sense to pad the last frame to the big frame_size,
so allow encoders to set the number of samples that they want
the last frame to be padded to instead.
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@outlook.com>
In particular, check that there is only one small last frame
in case the encoder has the AV_CODEC_CAP_SMALL_LAST_FRAME set.
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@outlook.com>
It is only used by encoders; in fact, AVCodecContext.time_base
is only used by encoders, so it is only useful for encoders.
Also constify the AVCodecContext parameter in it.
Also fixup the other headers a bit while removing now unnecessary
internal.h inclusions.
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@outlook.com>
Decoder-only, as the dimensions are set by the user when encoding.
Also fixup the other headers a bit while removing unnecessary internal.h
inclusions.
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@outlook.com>
Only used by decoders (encoders have ff_encode_alloc_frame()).
Also clean up the other headers a bit while removing now redundant
internal.h inclusions.
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@outlook.com>
Only used by decoders.
Also clean up the headers a bit while removing now unnecessary
internal.h inclusions.
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@outlook.com>
Handling this in general code makes more sense than handling it in
individual codec files, because it would be a lot of unnecessary code
duplication for the plenty of formats that support exporting ICC
profiles (jpg, png, tiff, webp, jxl, ...).
encode.c and decode.c will be in charge of initializing this state as
needed, so we merely need to make sure to uninit it afterwards from the
common destructor path.
Signed-off-by: Niklas Haas <git@haasn.dev>
The amount of padding samples reported by containers take into account the
extended samplerate in HE-AAC.
Fixes ticket #9671.
Signed-off-by: James Almer <jamrial@gmail.com>
The general decoding API uses bitstream filters and an AVFifo
and therefore AVCodecInternal contains pointers to an AVBSFContext
and to an AVFifo and lavc/internal.h includes lavc/bsf.h and
lavu/fifo.h.
Yet actually, only two files are supposed to use these, namely
avcodec.c and (mainly) decode.c. For all the other files,
it should be an opaque type that they should not touch and that
they need not know anything about. This can be achieved by not
including these headers and using the structs instead of the
corresponding typedefs.
This also forces translation units that really use the BSF
and the FIFO APIs themselves to include the relevant headers
directly instead of relying on indirect inclusions (up until now,
even avcodec.c and decode.c relied on fifo.h to be included
by internal.h).
Of course, it also avoids unnecessary rebuilds when bsf.h or fifo.h
change.
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@outlook.com>
It is a more fitting place for them.
Also move the definition of ff_log2_run to mathtables.c.
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@outlook.com>
Up until now, codec.h contains both public and private parts
of AVCodec. This exposes the internals of AVCodec to users
and leads them into the temptation of actually using them
and forces us to forward-declare structures and types that
users can't use at all.
This commit changes this by adding a new structure FFCodec to
codec_internal.h that extends AVCodec, i.e. contains the public
AVCodec as first member; the private fields of AVCodec are moved
to this structure, leaving codec.h clean.
Reviewed-by: Anton Khirnov <anton@khirnov.net>
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@outlook.com>
Also move FF_CODEC_TAGS_END as well as struct AVCodecDefault.
This reduces the amount of files that have to include internal.h
(which comes with quite a lot of indirect inclusions), as e.g.
most encoders don't need it. It is furthemore in preparation
for moving the private part of AVCodec out of the public codec.h.
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@outlook.com>
Since the request_channel_layout is used only by a handful of codecs,
move the option to codec private contexts.
Signed-off-by: Anton Khirnov <anton@khirnov.net>
Signed-off-by: James Almer <jamrial@gmail.com>
This is by definition the appropriate place for it.
Remove all the now unnecessary libavcodec/internal.h inclusions;
also remove other unnecessary headers from the affected files.
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@outlook.com>
This function is quite small (96B with GCC 11.2 on x64 Ubuntu 21.10
at -O3), making it more economical to duplicate it into libavformat
instead of exporting it as avpriv: Doing so saves 2x24B in .dynsim,
2x16B in .dynstr, 2x2B .gnu.version, 24B in .rela.plt, 16B in .plt,
16B in .plt.sec (if enabled), 4B .gnu.hash; besides the actual
duplicated code this also adds 2x8B .eh_frame_hdr and 24B .eh_frame.
In other words: Duplicating is neutral size-wise (it is also presumed
neutral for other systems). Given that it avoids the runtime
overhead of dynamic symbols, it is advantageouos to duplicate the
function.
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@outlook.com>
Decoders implementing the receive_frame API currently mostly use
stack packets to temporarily hold the packet they receive from
ff_decode_get_packet(). This role directly parallels the role of
in_pkt, the spare packet used in decode_simple_internal for the
decoders implementing the traditional decoding API. Said packet
is unused by the generic code for the decoders implementing the
receive_frame API, so allow them to use it to fulfill the function
it already fulfills for the traditional API for both APIs.
There is only one caveat in this: The packet is automatically
unreferenced in avcodec_flush_buffers(). But this is actually
positive as it means the decoders don't have to do this themselves
(in case the packet is preserved between receive_frame calls).
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@outlook.com>
Currently, the AV_PKT_FLAG_KEY is automatically set for audio encoders;
yet this is wrong, as both MLP and TrueHD have non-keyframes. So set it
based upon AV_CODEC_PROP_INTRA_ONLY (from the corresponding
AVCodecDescriptor) instead. This also sets it for some video codecs,
which is intended.
Reviewed-by: James Almer <jamrial@gmail.com>
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@outlook.com>
It is also used by libavfilter and it is only natural to define it
alongside ff_dlog().
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@outlook.com>
Decoders like cuviddec ignore and overwrite all the properties set by the generic
code as derived from AVCodecInternal.last_pkt_props. This flag ensures libavcodec
will not store and potentially queue input packets that ultimately will not be used.
Signed-off-by: James Almer <jamrial@gmail.com>
Up until now, ff_alloc_packet2() has a min_size parameter:
It is supposed to be a lower bound on the final size of the packet
to allocate. If it is not too far from the upper bound (namely,
if it is at least half the upper bound), then ff_alloc_packet2()
already allocates the final, already refcounted packet; if it is
not, then the packet is not refcounted and its data only points to
a buffer owned by the AVCodecContext (in this case, the packet will
be made refcounted in encode_simple_internal() in libavcodec/encode.c).
The goal of this was to avoid data copies and intermediate buffers
if one has a precise lower bound.
Yet those encoders for which precise lower bounds exist have recently
been switched to ff_get_encode_buffer() (which automatically allocates
final buffers), leaving only two encoders to actually set the min_size
to something else than zero (namely aliaspixenc and hapenc). Both of
these encoders use a very low lower bound that is not helpful in any
nontrivial case.
This commit therefore removes the min_size parameter as well as the
codepath in ff_alloc_packet2() for the allocation of final buffers.
Furthermore, the function has been renamed to ff_alloc_packet() and
moved to encode.h alongside ff_get_encode_buffer().
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@outlook.com>
Andreas Rheinhardt
Mark Thompson
James Almer
Anton Khirnov
Niklas Haas
Vittorio Giovara