ffmpeg and ffprobe in electron
Last updated
Last updated
using modified version of static ffmpeg /ffprobe lib.
autoEdit uses ffmpeg under the hood, and getting ffmpeg and electron to work can sometimes be problematic when setting up a new app, so about how this setup works in autoEdit, with simplified example.
I've since abstracted and published these two forks as npm packages
First things first let’s introduce the components will need, to make sure you are familiar with it.
A complete, cross-platform solution to record, convert and stream audio and video.
Very powerful go to option for open source manipulation of audio and video files.
ffprobe gathers information from multimedia streams and prints it in human- and machine-readable fashion.
It basically reads metadata information from audio and video files.
Build cross platform desktop apps with JavaScript, HTML, and CSS.
It basically allows you to package your app wit a custom version of the V8 engine which powers chrome to make it self contained, and if you wish, with the right adjustments, cross platform compatible.
ffmpeg-staticffmpeg static binaries for Mac OSX and Linux and Windows
ffprobe-staticStatic binaries for ffprobe
Is what we’ll use for packing our code base into an electron app that can be built and distributed to users. For example, for mac os x it can create a dmg or zip file to distrubute the app.
However, in context like electron, you don’t want to go and interfere with the user's systems configurations too much. For example what if they have another installation of ffmpeg that is slightly different and doesn’t have the dependencies you might need for the one in your app?
So we don’t want to either assume ffmpeg/ffprobe that is installed nor install it on their system as it could lead to conflicts and confusion. Nor do we want to make it as a requirement for the users to have that specific version we might need installed, as ffmpeg/ffprobe can be complicated to install and troubleshoot.
It’s very simple, you give it an audio or video file and it returns a plain text automated transcription.
On the plus side this is a pretty straightforward implementation but some of its limitations are that it does not account for cross-platform distribution eg if you are making if for Mac Linux and PC, or even different type of architecture, such as 64 or 32 bits if you were to support both newer and older computers. To do so you’d need a way to recognize the os and architecture and use the right binary.
Let’s consider a more involved example that gives more flexibility.
With this overview in mind let's look at how the ffmpeg + electron integration was done in this app.
static-ffmpeg module contains pre-built binaries for ffmpeg for the various operating system.
This gives you the advantage that you could be developing this app on Mac, Linux, and windows, without ffmpeg on your machine and could still be developing contributing to each OS version (with a few tweaks to the build process depending on the OS).
The other advantage is that if you wish you can change the ffmpeg binary that you want to ship with your electron app with a version that might have a specific set of dependencies.
In the case of autoEdit, it is a mac only app, but I tried to keep option to make windows and Linux version open as much as possible where it didn’t require a lot of extra effort to avoid it from becoming too much of an undertaking should I decide to do that in the future.
This means that if we want to use electron-builder and ffmpeg-static to have consistent file path to the ffmpeg binary we are packing inside electron both for providing a file path to use with fluent-ffmpeg in our code base, as well as for including the right binary for the right os distribution and excluding the rest we need the os name for mac os x to be consistent.
The problem with this setup is that we don’t want to have the binary for windows and Linux in the mac distribution as that would take up a lot of space. And we ideally want to keep our electron app as lightweight as possible.
In order to do that we have to add some logic to our build script to only package the binary for the os that we are building for.
Where the ${os} and ${arch} variable give you operating system(Mac, Linux, pc) and architecture (32 or 64 bits).
That's it, hope this gives you an idea of how to go about adding ffmpeg in your electron app.
I was recently asked about how to use with the in , and I had some notes but since they were at some sort of at draft stage I decided to expand more on this here, to make a more clear and comprehensive explanation.
I am going to use the code from the implementation of as well as an earlier "transcriber prototype" as a concrete example.
For those not familiar with it autoEdit is a text based video editing app for mac os x, , andfor more details.
It’s a library that wraps around the ffmpeg binary and provides an easier interface to run ffmpeg commands while in node. .
In this context, I’ll describe/explain in more details later.
Equivalent for ffprobe as described above for ffmpeg. And as above In this context, I’ll describe/explain in more details later.
A complete solution to package and build a ready for distribution Electron app with "auto update" support out of the box
As a side note, if you are using github, are a great way to host the packaged app distribution. See example where if you click on "Download Os X app" it
Last but not least see if you are not familiar with it or need a refresher.
When using fluent-ffmpeg, ffmpeg (and ffprobe) normally the library assumes that the binary is installed and.
. And that is our preferred choice for packaging ffmpeg inside electron.
to test whether users would accept automatically transcribed text, before building the first version of autoEdit 2 on top of IBM Watson STT service.
Disclaimer: It was a throwaway project to test that hypothesis . However, these have been addressed and improved in .
This project also uses (formerly "node web-kit") while autoEdit was later refactored to use electron. But for the purpose of adding support for ffmpeg, this is irrelevant for now.
Back to our point, in this use case, when working with STT a common problem is converting audio or video to the audio file format that meets the specs of the STT API. In this working prototype, I had a module to do just that .
As you can see we are setting the path to the ffmpeg binary.
Where where the ffmpeg binary for Mac OS X is located in the repository. When packaging/building the app for distribution this binary will be included with it.
In the audio/video conversion to obtain an audio that meets the STT API specs is a bit more involved and handled with a series of components.
Without getting too deep into the low-level implementation, there's an module that uses other components to create a transcription, read metadata, and create a video preview. It delegates the communication with the STT APIs to the component. Which uses other modules to split the media into 5 minutes chunks to speed up the transcription time. Such as the as part of this process.
As you might have noticed and . And to do so it’s relying on an external module. and .
The key reason why I ended up and creating my own version for simplicity is that as we’ll see in later section, when packaging the app, can recognise the operating system and architecture of the machine you are on,which is useful for including /excluding the binaries for that distribution.
But when electron-builder recognizes a Mac computer it recognizes it as the string mac. While static-ffmpeg uses the module which recognises a Mac computer as the string darwin. ()
So In my version of static-ffmpeg I . And .
And as you can see
To best understand the ffmpeg-staticmodule is using the module and .
I did something equivalent for my version of .
However even tho . Pull request welcome, should you have a use case where support for Linux is needed and want to contribute to this.
As an example, we have seen a whirlwind overview of the autoEdit architecture for the transcription and video conversion part at the beginning of this section. As you might remember there’s a module that .
While the module that requires the ffmpeg path is the which requires it from the config module.
And then passes it which then gets passed down the chain all the way to our trimmer module we saw in the previous section.
Using there is a files notation that can be used to specify these rules in .
In you can set the files to include/exclude as follows.
More details on this in .
If you have any thoughts, questions, ideas, alternatives, get in touch via email or twitter .
