autoEdit 2 Documentation
1.0.13
1.0.13
  • Introduction
  • Overview
    • Intro
      • Development approach
      • from 1.0.5 to 1.0.6
    • Architecture Overview
      • Lo fi Design Sketches
      • High fidelity sketches in HTML
    • Support the project
    • documentation section template
    • Build / Deployment
      • Deployment/build for Mac OS X
      • Deployment / Build for Linux
      • Deployment / Build for Windows
      • Travis CI continuous build
        • previous travis setup
  • R&D Doc
    • About R&D doc section
    • Transcription & Media Processing
      • Transcription json
      • Transcriber
        • audio to video
        • STT sdks
          • IBM Watson STT
          • Gentle STT
          • Pocketsphinx
      • Video preview conversion
      • Read metadata
    • Hypertranscript
    • Selections, Annotations, Papercuts
      • Selections
      • Annotations
      • Tags
    • Paper-edit
      • Paper-edit json
      • search-filter
      • drag-and-drop
      • Preview Paper-edit video
    • Export
      • EDL export
      • XML export
      • mp4 export
  • Appendix
    • Dev configuration
    • Current db setup
    • EDL Format
    • Reusable components
    • Prerequisites
    • Testing
    • Updating automated documentation
    • ffmpeg and ffprobe in electron
    • Adding STT services
  • Appendix - Data structures
    • IBM Watson json specs
    • Gentle Json transcription specs
    • Pocketsphinx results
    • autoEdit transcription Json
  • QA List
    • QA Intro
  • Adobe Panel
    • autoEdit Adobe CEP Panel dev setup
      • Adobe Extensions HostList codes
    • autoEdit Adobe CEP Panel integration overview
    • Jsx functions for Adobe CEP autoEdit adobe Panel
    • Packaging and distributing Adobe CEP Extensions
      • Packaging signing Adobe CEP Panel in details
      • Submit to Adobe
  • Project Page
    • Build project page
    • Build/update demo front end page
  • Roadmap
    • Improvements
    • Roadmap
      • Paper-editing Roadmap
      • Extra Features Roadmap
      • Future Roadmap
        • Live video editing
        • Social Media Export
        • Translate transcriptions
        • Web app
          • Multi-user collaboration
        • Searchable Editable Archive
        • NLP insights
        • Slack/Chat bot integration
        • Interactive dev tool
        • Phone mms integration with twillio
        • B-roll computational photography
Powered by GitBook
On this page
  • High level Overview
  • High level overview of the parts
  • Data structures
  • Stack
  • electron
  • Backbone app
  • Backbone.sync
  • lib/interactive_transcription_generator
  • edl_composer module
  • ffmpeg and electron
  • STT
  • srtmodule
  • Tests folder
  • Other docs
  1. Overview

Architecture Overview

Previousfrom 1.0.5 to 1.0.6NextLo fi Design Sketches

Last updated 6 years ago

High level Overview

  1. autoEdit uses Speech to text services to create automatic transcription from a video or audio file

  2. The user can make text selections

  3. Export those selections as a video sequence in the editing software of choice

See for more details. And project on github.

High level overview of the parts

It's helpful to discuss the implementation of autoEdit by looking at the parts that make up the whole as stand alone overarching components, that are built of other smaller components and modules.

Tick (✔) for features implemented in 1.0.12

Data structures

Main data structures in the project

  • Transcription json

  • paper-cuts json

  • paper-edit json

Stack

It is built using node electron and backbone.

electron

Code of the electron app is in the ./electron folder. The backbone app is added to ./electron/index.html with browserify using npm scripts.

Backbone app

The backbone app is in lib/app folder. For troubleshooting you can use cmd+ alt +i to get the electron developer console. There you have appTranscription and appPaperedit two backbone routers in the global window scope that give you access to getting to individual backbone models and collections for transcriptions and paper-edit.

Backbone.sync

in index.html the window object is used to provide an interface between the electron client side code code packaged with browserify and the 'backend' that can make the file system calls in node.

If you use require in the html file, then you are in node context, and can use module like fs but if you use a script tag, then you are in js client side code and don't have access to thos function.

The downside of being in node context on the client side, without using a bundle like browserify and requring modules directly is that the front end is no longer portable as standalone client side web app if needed, eg for the demo page.

// if require is not undefined then we are in node context in `index.html` , and therefore using 
if (typeof require !== 'undefined') {
    // other code here..
    window.DB = require('./db.js');
    //...

in lib/app/app.js the choice between the demo db and the production db is made.

// Connect up the backend for backbone
if (typeof window.DB !== 'undefined') {
  Backbone.sync = window.DB;
} else {
  Backbone.sync = require('./demo_db');
}

demo_paperedit.json anddemo_transcription.json provide the data for the demo when index.html is run in client side mode in the browser. and lib/app/demo_db.js provides the logic for the demo db.

DB in ./electron/db.js allows to connect the backbone front to the medeadown/ linvodb3 databse locally. As well as trigger ./lib/interactive_transcription_generator component to at a high level

  • read metadata of the media file, for the EDL sequence

  • create video preview

  • create audio version to send to STT APIs

lib/interactive_transcription_generator

At a hight level this module:

  • converts the file to an audio file meet the STT Specs, eg for IBM Watson, using the submodule transcriber.

  • for IBM Watson only if greater then 5 min long it splits it into 5 min chunk.

    • It keeps tracks of the time offset of each clip.

  • It sends audio files to STT API.

    • for IBM Watson only If submitted file was greater then 5 min.

    • for IBM Watson only When results starts to come back as json after about 5 min or less results are re-interpolated into one json file.

  • The json returned by STT Service API is converted into json meeting autoEdit2 specs and saved in db.

  • User can now view interactive an transcription.

The transcriber module used by interactive_transcription_generator can also chose between using Gentle open source STT, Pocketsphinx or IBM to generate the transcription depending on what was specified by the user.

interactive_transcription_generator folder structure overview:

.
├── README.md
├── index.js
├── transcriber
│   ├── convert_to_audio.js
│   ├── gentle_stt_node
│   │   ├── gentle_stt.js
│   │   ├── index.js
│   │   └── parse_gentle_stt.js
│   ├── ibm_stt_node
│   │   ├── parse.js
│   │   ├── sam_transcriber_json_convert.js
│   │   ├── send_to_watson.js
│   │   └── write_out.js
│   ├── index.js
│   ├── pocketsphinx
│   │   ├── README.md
│   │   ├── index.js
// pocketsphinx binaries 
│   │   ├── pocketsphinx
│   │   ├── pocketsphinx.js
│   │   ├── pocketsphinx_converter.js
// pocketsphinx binaries 
│   │   ├── sphinxbase
│   │   └── video_to_audio_for_pocketsphinx.js
│   ├── split.js
│   └── trimmer.js
├── video_metadata_reader
│   └── index.js
└── video_to_html5_webm
    └── index.js

edl_composer module

ffmpeg and electron

I've since abstracted and published these two forks as npm packages

STT

The app uses the following Speech To Text systems to generate transcription

The user can then select text and export a video sequence to the video editing software of choice.

srtmodule

Tests folder

Future plans include refactoring, and separate tests to be closer to their components, as well as migrate from jasmine to jest.

Other docs

Is designed so that the front end in backbone can be used as standalone static site. Which is how is run, with an hard coded sample data in backbone.sync.

db.js overrides backbone.sync method to provide a backend for the app and persistent storage using linvodb3, which uses medeadown to storing db on the user file system. .

To re-cap After the user uploads a video or audio file the backbone app override default and calls the electron/db.js which after saving the transcription model in db, triggers this module to get stt transcription, video preivew, and metadata info.

interactive_transcription_generator. On top of prepping the audio or video file to get a transcription from IBM, it also generates a webm html5 video preview and reads the metadata, which is something needed to make an .

The pocketsphinx module was originally extracted from the . The implementation of this module is discussed in more details in subsequent sections.

See , and as well as .

This module as also been abstracted as separate and

autoEdit uses ffmpeg under the hood, and getting ffmpeg and electron to work can sometimes be problematic when setting up a new app, so about how this setup works in autoEdit, with simplified example.

See for more details.

The module used to export transcriptions as captions file is a simplified version of also .

Tests are in the spec folder, to run the test suite npm run test. . Testes are setup to be all in one place rather then divided with their respective components, for ease of use. Altho test coverage is far from complete and could do with some attention, s if that's something you'd be interested getting involved with.

There is also a google doc for a , which was original used with Vox Media QA team. It is now not up to date to the paper-editing features, but is comprehensive enough for the test. And a great starting point to make a new one.

There is also an that was used for version 1.0.6 to add the paper-editing functionality.

See user manual
README
the demo
See current db setup tutorial for more info
backbone.sync
EDL
electron version of the Video grep project
EDL format
Export section
EDL composer module
npm module
github repository
I wrote here
ffmpeg-static-electron
ffprobe-static-electron
this section
srt module
srt parse composer module
npm
Uses jasmine for testing
ee supporting the project
"QA Test Plan autoEdit 2"
"R&D planning google doc"