Video thumbnails

[Robert Helling]

Hi,

first of all thanks to everybody who was on yesterday’s video call. That was both useful and any enjoyed seeing you (many of you for the first time)!

I had some more thoughts regarding the discussion of how to generate thumbnails for videos. After hearing how difficult that seems to be to do within Qt and all kinds of licensing problems with even shipping the necessary codecs it seems to me there is no way around using external tools. For playback just rely on the OS to playback a video (just as we do for opening images).

To produce thumbnails (which seems to be even more urgent) I looked again into ffmpeg and that appears to be the tool of choice: It is available for all relevant operating systems, many users probably already have it installed (e.g. as part of VLC) and if not installing it is pretty straight forward (if no patents prevent you to do so but that would not be our problem). We would simply have a preference with the path to the ffmpeg binary. Then (maybe triggered by the user) we could run a command like

ffmpeg -i myvideo.avi -vf fps=1/60 img%03d.jpg

(for more details see https://trac.ffmpeg.org/wiki/Create%20a%20thumbnail%20image%20every%20X%20seconds%20of%20the%20video )/ in a directory per video file and then take those thumbnails as a representation of the video inside subsurface. This would have to done only once per video, the rest would be treated just like the image we already have.

This solution sounds to me both sufficiently user friendly as well as straight forward to implement.

What do you think?

Best
Robert

--
.oOo.oOo.oOo.oOo.oOo.oOo.oOo.oOo.oOo.oOo.oOo.oOo.oOo.oOo.oOo.oOo.oOo.oOo.oOo.oO
Robert C. Helling Elite Master Course Theoretical and Mathematical Physics
Scientific Coordinator
Ludwig Maximilians Universitaet Muenchen, Dept. Physik
Phone: [phone] Theresienstr. 39, rm. B339
http://www.atdotde.de

Enhance your privacy, use cryptography! My PGP keys have fingerprints
A9D1 A01D 13A5 31FA 6515 BB44 0820 367C 36BC 0C1D and
DCED 37B6 251C 7861 270D 5613 95C7 9D32 9A8D 9B8F

[Berthold Stoeger]

Hi Robert,

Post by Robert Helling
I had some more thoughts regarding the discussion of how to generate
thumbnails for videos. After hearing how difficult that seems to be to do
within Qt and all kinds of licensing problems with even shipping the
necessary codecs it seems to me there is no way around using external
tools. For playback just rely on the OS to playback a video (just as we do
for opening images).
To produce thumbnails (which seems to be even more urgent) I looked again
into ffmpeg and that appears to be the tool of choice: It is available for
all relevant operating systems, many users probably already have it
installed (e.g. as part of VLC) and if not installing it is pretty straight
forward (if no patents prevent you to do so but that would not be our
problem). We would simply have a preference with the path to the ffmpeg
binary. Then (maybe triggered by the user) we could run a command like
ffmpeg -i myvideo.avi -vf fps=1/60 img%03d.jpg

That is indeed my plan B - given that Qt turned out to be not nearly as
helpful as I would have expected / hoped. As I have mentioned in a PR, my next
attempt will be using the ffmpeg project's LGPL libraries libavcodec and
libavformat but we should definitely also consider the option of calling an
external application. That might be less of a hassle.

Concerning shipping with either ffmpeg or one of its libraries, they provide a
mini-FAQ here:

https://www.ffmpeg.org/legal.html

Not sure what to make of that (since we're neither private nor commercial
users).

Berthold

[Dirk Hohndel]

Post by Berthold Stoeger

Post by Robert Helling
To produce thumbnails (which seems to be even more urgent) I looked again
into ffmpeg and that appears to be the tool of choice: It is available for
all relevant operating systems, many users probably already have it
installed (e.g. as part of VLC) and if not installing it is pretty straight
forward (if no patents prevent you to do so but that would not be our
problem). We would simply have a preference with the path to the ffmpeg
binary. Then (maybe triggered by the user) we could run a command like
ffmpeg -i myvideo.avi -vf fps=1/60 img%03d.jpg

That is indeed my plan B - given that Qt turned out to be not nearly as
helpful as I would have expected / hoped. As I have mentioned in a PR, my next
attempt will be using the ffmpeg project's LGPL libraries libavcodec and
libavformat but we should definitely also consider the option of calling an
external application. That might be less of a hassle.

The advantage of calling an app vs. linking against the libraries is that for
calling an app we need to tell the user how to do that, vs. for shipping the
libraries we need to build them on all the platforms, integrate them into our
package and deal with the potential licensing / patent pain created by that.
You can guess which way I'm leaning here :-)

Post by Berthold Stoeger
Concerning shipping with either ffmpeg or one of its libraries, they provide a
https://www.ffmpeg.org/legal.html
Not sure what to make of that (since we're neither private nor commercial
users).

Yes, that's part of the problem. While typically the license holders don't go
after "small projects" - if they decided to do so, the person creating the
binary distribution and providing it for download is the likely target of such
legal action. And that makes me even more excited about a solution where
all we do is call external software that we didn't ship.

/D

[Berthold Stoeger]

Post by Dirk Hohndel
The advantage of calling an app vs. linking against the libraries is that
for calling an app we need to tell the user how to do that, vs. for
shipping the libraries we need to build them on all the platforms,
integrate them into our package and deal with the potential licensing /
patent pain created by that. You can guess which way I'm leaning here :-)

Make it easy for the user, what else...? ;)

But seriously, let's land #1463 first. In a private branch I already have
connected this to a "VideoFrameExtractor". I just need to replace QMultimedia
by ffmpeg. Whether we use an executable [Windows is posix, so it does know the
concept of stdout, right?] or a library is then just a minor implementation
detail.

But give me a week or two, please.

Berthold

[Dirk Hohndel]

Post by Berthold Stoeger

Post by Dirk Hohndel
The advantage of calling an app vs. linking against the libraries is that
for calling an app we need to tell the user how to do that, vs. for
shipping the libraries we need to build them on all the platforms,
integrate them into our package and deal with the potential licensing /
patent pain created by that. You can guess which way I'm leaning here :-)

Make it easy for the user, what else...? ;)
But seriously, let's land #1463 first. In a private branch I already have
connected this to a "VideoFrameExtractor". I just need to replace QMultimedia
by ffmpeg. Whether we use an executable [Windows is posix, so it does know the
concept of stdout, right?] or a library is then just a minor implementation
detail.
But give me a week or two, please.

Just to confirm - you don't consider #1463 ready to be pulled, you want to work
on it for a week or two, correct?

/D

[Berthold Stoeger]

Post by Dirk Hohndel
Just to confirm - you don't consider #1463 ready to be pulled, you want to
work on it for a week or two, correct?

The time frame was with respect to including ffmpeg. But yes, I will polish
#1463 a bit more and will notify you when I consider it ready.

Berthold

[Robert Helling]

Berthold,

Post by Berthold Stoeger
Make it easy for the user, what else...? ;)
But seriously, let's land #1463 first. In a private branch I already have
connected this to a "VideoFrameExtractor". I just need to replace QMultimedia
by ffmpeg. Whether we use an executable [Windows is posix, so it does know the
concept of stdout, right?] or a library is then just a minor implementation
detail.
But give me a week or two, please.

I might to have some time over the weekend for coding. Would you mind sharing what you already have by pushing that branch to your GitHub account? I would then try to start hooking up ffmpeg.

My idea for how it works would be something like:

From the images tab, after associating a video to a dive, you can (maybe by a context menu entry) trigger the generation of thumbnails. For those, I would create a directory

videofile.mpeg.thumbnails

and create image files there.

From there, Subsurface could look for such a directory and then display the thumbnails as images (at the given time intervals). So the idea would be to trade disk space (people who have videos should have plenty of that) for runtime generating the thumbnails on the fly for each run of Subsurface.

What do you think?

Best
Robert
--
.oOo.oOo.oOo.oOo.oOo.oOo.oOo.oOo.oOo.oOo.oOo.oOo.oOo.oOo.oOo.oOo.oOo.oOo.oOo.oO
Robert C. Helling Elite Master Course Theoretical and Mathematical Physics
Scientific Coordinator
Ludwig Maximilians Universitaet Muenchen, Dept. Physik
Phone: [phone] Theresienstr. 39, rm. B339
http://www.atdotde.de

Enhance your privacy, use cryptography! My PGP keys have fingerprints
A9D1 A01D 13A5 31FA 6515 BB44 0820 367C 36BC 0C1D and
DCED 37B6 251C 7861 270D 5613 95C7 9D32 9A8D 9B8F

[Berthold Stoeger]

Hi Robert,

Post by Robert Helling
I might to have some time over the weekend for coding. Would you mind
sharing what you already have by pushing that branch to your GitHub
account? I would then try to start hooking up ffmpeg.

You can see what I have in my branch video6, but it will not be very useful,
owing to the thread peculiarities of QMediaPlayer (you have to start it in the
UI thread).

My intention was that instead of calling the UI thread with

QMetaObject::invokeMethod(VideoFrameExtractor::instance(), "extract",
Qt::AutoConnection, Q_ARG(QString, originalFilename), Q_ARG(QString,
filename));

run a thread with 'QtConcurrent::run' which then uses signal/slots to send the
extracted thumbnail.

Post by Robert Helling
From the images tab, after associating a video to a dive, you can (maybe by
a context menu entry) trigger the generation of thumbnails. For those, I
would create a directory

We already do all that. And the code should also recognize if thumbnails are
older than the file. So I'm not sure if implementing a parallel thumbnail-
system for videos is a UI improvement.

Post by Robert Helling
videofile.mpeg.thumbnails
and create image files there.
From there, Subsurface could look for such a directory and then display the
thumbnails as images (at the given time intervals). So the idea would be to
trade disk space (people who have videos should have plenty of that) for
runtime generating the thumbnails on the fly for each run of Subsurface.
What do you think?

Can't ffmpeg write the image to stdout? Then we could just read it in a thread
and send it via signal/slot to the thumbnailing system. I always prefer "push"
over "pull" interfaces.

I'm off to a concert-weekend with probably limited internet access. So I won't
be able to do much on weekend.

Berthold

[Robert Helling]

Berthold,

Post by Berthold Stoeger
We already do all that. And the code should also recognize if thumbnails are
older than the file. So I'm not sure if implementing a parallel thumbnail-
system for videos is a UI improvement.

Post by Robert Helling
videofile.mpeg.thumbnails
and create image files there.
From there, Subsurface could look for such a directory and then display the
thumbnails as images (at the given time intervals). So the idea would be to
trade disk space (people who have videos should have plenty of that) for
runtime generating the thumbnails on the fly for each run of Subsurface.
What do you think?

Can't ffmpeg write the image to stdout? Then we could just read it in a thread
and send it via signal/slot to the thumbnailing system. I always prefer "push"
over "pull" interfaces.

I don’t know about writing to stdout, maybe. But still those would be several, wouldn’t it? What do you think is a good rate, one image per minute?

Let me modify my proposal: Trigger the generation upon opening the dive (and not having thumbnails) in a background task. Store the images in a temporary directory. Upon completion of the ffmpeg process add those images to the Subsurface thumbnails. Does that sound better (that would be a signal based push if you like).

Best
Robert

--
.oOo.oOo.oOo.oOo.oOo.oOo.oOo.oOo.oOo.oOo.oOo.oOo.oOo.oOo.oOo.oOo.oOo.oOo.oOo.oO
Robert C. Helling Elite Master Course Theoretical and Mathematical Physics
Scientific Coordinator
Ludwig Maximilians Universitaet Muenchen, Dept. Physik
Phone: [phone] Theresienstr. 39, rm. B339
http://www.atdotde.de

Enhance your privacy, use cryptography! My PGP keys have fingerprints
A9D1 A01D 13A5 31FA 6515 BB44 0820 367C 36BC 0C1D and
DCED 37B6 251C 7861 270D 5613 95C7 9D32 9A8D 9B8F

[Stöger, Berthold]

Hi Robert,

I would just calculate one thumbnail at a time. But just go wild and I can try to marry our solutions later, if needed.

Sending via webmail, so the mailinglist will probably reject.

Berthold

________________________________________
From: Robert Helling [***@lmu.de]
Sent: Friday, July 6, 2018 3:38 PM
To: Stöger, Berthold
Cc: Dirk Hohndel; ***@subsurface-divelog.org
Subject: Re: Video thumbnails

Berthold,

Post by Berthold Stoeger
We already do all that. And the code should also recognize if thumbnails are
older than the file. So I'm not sure if implementing a parallel thumbnail-
system for videos is a UI improvement.

Post by Robert Helling
videofile.mpeg.thumbnails
and create image files there.
From there, Subsurface could look for such a directory and then display the
thumbnails as images (at the given time intervals). So the idea would be to
trade disk space (people who have videos should have plenty of that) for
runtime generating the thumbnails on the fly for each run of Subsurface.
What do you think?

Can't ffmpeg write the image to stdout? Then we could just read it in a thread
and send it via signal/slot to the thumbnailing system. I always prefer "push"
over "pull" interfaces.

I don’t know about writing to stdout, maybe. But still those would be several, wouldn’t it? What do you think is a good rate, one image per minute?

Let me modify my proposal: Trigger the generation upon opening the dive (and not having thumbnails) in a background task. Store the images in a temporary directory. Upon completion of the ffmpeg process add those images to the Subsurface thumbnails. Does that sound better (that would be a signal based push if you like).

Best
Robert

--
.oOo.oOo.oOo.oOo.oOo.oOo.oOo.oOo.oOo.oOo.oOo.oOo.oOo.oOo.oOo.oOo.oOo.oOo.oOo.oO
Robert C. Helling Elite Master Course Theoretical and Mathematical Physics
Scientific Coordinator
Ludwig Maximilians Universitaet Muenchen, Dept. Physik
Phone: [phone] Theresienstr. 39, rm. B339
http://www.atdotde.de

Enhance your privacy, use cryptography! My PGP keys have fingerprints
A9D1 A01D 13A5 31FA 6515 BB44 0820 367C 36BC 0C1D and
DCED 37B6 251C 7861 270D 5613 95C7 9D32 9A8D 9B8F

[Berthold Stoeger]

Hi Robert,

Post by Stöger, Berthold

Post by Berthold Stoeger
Can't ffmpeg write the image to stdout? Then we could just read it in a
thread and send it via signal/slot to the thumbnailing system. I always
prefer "push" over "pull" interfaces.

I don’t know about writing to stdout, maybe. But still those would be
several, wouldn’t it? What do you think is a good rate, one image per
minute?
Let me modify my proposal: Trigger the generation upon opening the dive (and
not having thumbnails) in a background task. Store the images in a
temporary directory. Upon completion of the ffmpeg process add those images
to the Subsurface thumbnails. Does that sound better (that would be a
signal based push if you like).

I pushed a very rudimentary PR to github, which shows how this could be
married with the current thumbnailing system. In a first test on Linux, I was
amazed how nicely this works!

But it is of course still very rough. ffmpeg muse be in the current path. The
difficult case will be what to do if no ffmpeg is installs or it fails. I
think we shouldn't show a "broken" icon in order not to penalize users without
ffmpeg. But should we add an entry into the thumbnail cache, etc?

I can dig up the my old code that added a watermark to the thumbnails to
signal "video file".

Hope that is useful,

Berthold