ffmpeg and ffprobe in electron
Last updated
Last updated
using modified version of static ffmpeg /ffprobe lib.
autoEdit uses ffmpeg under the hood, and getting ffmpeg and electron to work can sometimes be problematic when setting up a new app, so I wrote here about how this setup works in autoEdit, with simplified example.
I've since abstracted and published these two forks as npm packages
I was recently asked about how to use ffmpeg with the fluent-ffmpeg node library in electron, and I had some notes in the autoEdit documentation but since they were at some sort of at draft stage I decided to expand more on this here, to make a more clear and comprehensive explanation.
I am going to use the code from the implementation of autoEdit.io as well as an earlier "transcriber prototype" as a concrete example.
For those not familiar with it autoEdit is a text based video editing app for mac os x, check out the website , README and user manual for more details.
First things first let’s introduce the components will need, to make sure you are familiar with it.
A complete, cross-platform solution to record, convert and stream audio and video.
Very powerful go to option for open source manipulation of audio and video files.
ffprobe gathers information from multimedia streams and prints it in human- and machine-readable fashion.
It basically reads metadata information from audio and video files.
Build cross platform desktop apps with JavaScript, HTML, and CSS.
It basically allows you to package your app wit a custom version of the V8 engine which powers chrome to make it self contained, and if you wish, with the right adjustments, cross platform compatible.
It’s a library that wraps around the ffmpeg binary and provides an easier interface to run ffmpeg commands while in node. It also supports running commands for ffprobe.
ffmpeg-staticffmpeg static binaries for Mac OSX and Linux and Windows
In this context I used a modified version, I’ll describe/explain in more details later. From the original repository.
ffprobe-staticStatic binaries for ffprobe
Equivalent for ffprobe as described above for ffmpeg. And as above In this context I used a modified version, I’ll describe/explain in more details later. From the original repository.
A complete solution to package and build a ready for distribution Electron app with "auto update" support out of the box https://electron.build
Is what we’ll use for packing our code base into an electron app that can be built and distributed to users. For example, for mac os x it can create a dmg or zip file to distrubute the app.
As a side note, if you are using github, github releases are a great way to host the packaged app distribution. See autoEdit example where if you click on "Download Os X app" it takes you to the releases page.
Last but not least see here more details on using export/require for node modules if you are not familiar with it or need a refresher.
When using fluent-ffmpeg, ffmpeg (and ffprobe) normally the library assumes that the binary is installed and/or the path available as an environment variable, as described here.
However, in context like electron, you don’t want to go and interfere with the user's systems configurations too much. For example what if they have another installation of ffmpeg that is slightly different and doesn’t have the dependencies you might need for the one in your app?
So we don’t want to either assume ffmpeg/ffprobe that is installed nor install it on their system as it could lead to conflicts and confusion. Nor do we want to make it as a requirement for the users to have that specific version we might need installed, as ffmpeg/ffprobe can be complicated to install and troubleshoot.
Luckily is possible to set the path to ffmpeg and ffprobe binary. And that is our preferred choice for packaging ffmpeg inside electron.
This was a prototype to test whether users would accept automatically transcribed text, before building the first version of autoEdit 2 on top of IBM Watson STT service.
It’s very simple, you give it an audio or video file and it returns a plain text automated transcription.
Disclaimer: It was a throwaway project to test that hypothesis see issues in repo for reasons why it’s not distribution ready. However, these have been addressed and improved in autoEdit 2.
This project also uses nwjs (formerly "node web-kit") while autoEdit was later refactored to use electron. But for the purpose of adding support for ffmpeg, this is irrelevant for now.
Back to our point, in this use case, when working with STT a common problem is converting audio or video to the audio file format that meets the specs of the STT API. In this working prototype, I had a module to do just that video_to_audio.js.
As you can see at line 12 we are setting the path to the ffmpeg binary.
Where "./bin/ffmpeg" corresponds to the path where the ffmpeg binary for Mac OS X is located in the repository. When packaging/building the app for distribution this binary will be included with it.
On the plus side this is a pretty straightforward implementation but some of its limitations are that it does not account for cross-platform distribution eg if you are making if for Mac Linux and PC, or even different type of architecture, such as 64 or 32 bits if you were to support both newer and older computers. To do so you’d need a way to recognize the os and architecture and use the right binary.
Let’s consider a more involved example that gives more flexibility.
In autoEdit the audio/video conversion to obtain an audio that meets the STT API specs is a bit more involved and handled with a series of components.
Without getting too deep into the low-level implementation, there's an interactive_transcription_generator module that uses other components to create a transcription, read metadata, and create a video preview. It delegates the communication with the STT APIs to the transcriber component. Which uses other modules to split the media into 5 minutes chunks to speed up the transcription time. Such as the trimmer, module which trims a video or audio file. as part of this process.
With this overview in mind let's look at how the ffmpeg + electron integration was done in this app.
As you might have noticed the config module contains the path to ffmpeg and to ffprobe. And to do so it’s relying on an external module. ffmpeg-static and ffprobe-static.
static-ffmpeg module contains pre-built binaries for ffmpeg for the various operating system.
This gives you the advantage that you could be developing this app on Mac, Linux, and windows, without ffmpeg on your machine and could still be developing contributing to each OS version (with a few tweaks to the build process depending on the OS).
The other advantage is that if you wish you can change the ffmpeg binary that you want to ship with your electron app with a version that might have a specific set of dependencies.
In the case of autoEdit, it is a mac only app, but I tried to keep option to make windows and Linux version open as much as possible where it didn’t require a lot of extra effort to avoid it from becoming too much of an undertaking should I decide to do that in the future.
The key reason why I ended up forking the original and creating my own version for simplicity is that as we’ll see in later section, when packaging the app, electron-builder can recognise the operating system and architecture of the machine you are on,which is useful for including /excluding the binaries for that distribution.
But when electron-builder recognizes a Mac computer it recognizes it as the string mac. While static-ffmpeg uses the os module os.platform() which recognises a Mac computer as the string darwin. (os module also explained in more simple terms here)
This means that if we want to use electron-builder and ffmpeg-static to have consistent file path to the ffmpeg binary we are packing inside electron both for providing a file path to use with fluent-ffmpeg in our code base, as well as for including the right binary for the right os distribution and excluding the rest we need the os name for mac os x to be consistent.
So In my version of static-ffmpeg I changed the folder to the mac binary to be called mac instead of darwin. And added a bit of logic so that when os module recognizes it as darwin it gets assigned as mac.
And as you can see the same thing was done for windows to change from win32 to win for the same reason.
To best understand the ffmpeg-staticmodule is useful to look the part where the path to the binary is constructed using the path module and platform variable to construct the file path.
I did something equivalent for my version of ffprobe-static.
However what’s missing is the logic for the support for Linux binary in the index.js file even tho the binaries for Linux are in the repository. Pull request welcome, should you have a use case where support for Linux is needed and want to contribute to this.
As an example, we have seen a whirlwind overview of the autoEdit architecture for the transcription and video conversion part at the beginning of this section. As you might remember there’s a module that sets the ffmepg path, trimmer.js.
While the module that requires the ffmpeg path is the interactive_transcription_generator which requires it from the config module.
And then passes it as a config parameter to the transcriber module which then gets passed down the chain all the way to our trimmer module we saw in the previous section.
The problem with this setup is that we don’t want to have the binary for windows and Linux in the mac distribution as that would take up a lot of space. And we ideally want to keep our electron app as lightweight as possible.
In order to do that we have to add some logic to our build script to only package the binary for the os that we are building for.
Using electron-builder there is a files notation that can be used to specify these rules in package.json.
Where the ${os} and ${arch} variable give you operating system(Mac, Linux, pc) and architecture (32 or 64 bits).
In package.json you can set the files to include/exclude as follows.
More details on this in electron-builder documentation.
That's it, hope this gives you an idea of how to go about adding ffmpeg in your electron app.
If you have any thoughts, questions, ideas, alternatives, get in touch via email or twitter @pietropassarell.
