autoEdit 2 Documentation
1.0.13
1.0.13
  • Introduction
  • Overview
    • Intro
      • Development approach
      • from 1.0.5 to 1.0.6
    • Architecture Overview
      • Lo fi Design Sketches
      • High fidelity sketches in HTML
    • Support the project
    • documentation section template
    • Build / Deployment
      • Deployment/build for Mac OS X
      • Deployment / Build for Linux
      • Deployment / Build for Windows
      • Travis CI continuous build
        • previous travis setup
  • R&D Doc
    • About R&D doc section
    • Transcription & Media Processing
      • Transcription json
      • Transcriber
        • audio to video
        • STT sdks
          • IBM Watson STT
          • Gentle STT
          • Pocketsphinx
      • Video preview conversion
      • Read metadata
    • Hypertranscript
    • Selections, Annotations, Papercuts
      • Selections
      • Annotations
      • Tags
    • Paper-edit
      • Paper-edit json
      • search-filter
      • drag-and-drop
      • Preview Paper-edit video
    • Export
      • EDL export
      • XML export
      • mp4 export
  • Appendix
    • Dev configuration
    • Current db setup
    • EDL Format
    • Reusable components
    • Prerequisites
    • Testing
    • Updating automated documentation
    • ffmpeg and ffprobe in electron
    • Adding STT services
  • Appendix - Data structures
    • IBM Watson json specs
    • Gentle Json transcription specs
    • Pocketsphinx results
    • autoEdit transcription Json
  • QA List
    • QA Intro
  • Adobe Panel
    • autoEdit Adobe CEP Panel dev setup
      • Adobe Extensions HostList codes
    • autoEdit Adobe CEP Panel integration overview
    • Jsx functions for Adobe CEP autoEdit adobe Panel
    • Packaging and distributing Adobe CEP Extensions
      • Packaging signing Adobe CEP Panel in details
      • Submit to Adobe
  • Project Page
    • Build project page
    • Build/update demo front end page
  • Roadmap
    • Improvements
    • Roadmap
      • Paper-editing Roadmap
      • Extra Features Roadmap
      • Future Roadmap
        • Live video editing
        • Social Media Export
        • Translate transcriptions
        • Web app
          • Multi-user collaboration
        • Searchable Editable Archive
        • NLP insights
        • Slack/Chat bot integration
        • Interactive dev tool
        • Phone mms integration with twillio
        • B-roll computational photography
Powered by GitBook
On this page
  • Outline the problem
  • Areas of Interest
  • Reusable architecture and components
  • Video on the web
  • HCI: A Comprehensive Solution
  • Video Transcriptions
  • The Output
  • Structure of the documentation
  • R&D Doc
  • The Stack
  • Trello board
  • Gitbooks
  1. Overview

Intro

PreviousIntroductionNextDevelopment approach

Last updated 6 years ago

Working Draft R&D Documentation for Developers for 1.0.13

Outline the problem

For a video producer working on a factual/documentary production, editing video interviews is a time-consuming activity.

In the current work-flow, once the producer has the video file, they would have to find the quotes, which often entails manually scrubbing through the video in search of usable sound bites, which can be something like finding a needle in a haystack, particularly if the video is lengthy. Additionally, transcriptions are not a viable option either, as often there is no time to wait for them or no budget, and without time- codes, transcriptions do not help to speed up the process.

In a fast paced newsroom often there is no time for such a time-consuming process.

The aim of the autoEdit project is to produce an application that can maximise not only the depth of content but also the speed in which the content is produced.

The approach was to look at the traditional paper-editing workflow from documentary production, and see how that could translate into a digital world. .

Areas of Interest

One reason why this project is interesting is because it utilises the new possibilities that have opened up with the node ecosystem, that allow for great level of code reusability and the introduction of the HTML5 video tag. As well as the opportunities that have arisen in combining this with the increasing quality of speech to text technologies.

Reusable architecture and components

Combined with a mobile first approach, this allows to develop desktop, web, mobile version of the same app without the traditional overhead that this would involve.

This also lower the barrier of entry for this type of development.

Video on the web

HCI: A Comprehensive Solution

One of the challenges around this project is providing a comprehensive work-flow for the journalist and the user that is clear and easy to follow through from beginning to end, without requiring any training.

Video Transcriptions

The fastest way to be able to quickly select a caption from a video, is to search a transcription of that video. Therefore a challenge was to research and identify the most appropriate system to get the transcription of any given video. Human generated transcriptions are subject to time and resource constraints. While automated transcriptions need to meet a certain threshold of acceptance from the user, as well as have accurate timecodes in order to be searchable and in sync with the video for quick feedback.

The Output

Paper-editing it's only one stage in the documentary production when editing video interview, it generally corresponds to a rough cut stage, where a video sequence is assembled in a video editing software. This is then followed by a fine cut, where the video sequence is polished, cut-aways/B-roll shots are added to it etc..

It was therefore important for the output to integrate with video editing software.

This proved easier and more widely compatible to work with then an XML.

Structure of the documentation

This documentation is meant to serve as resource for the development of autoEdit. With the higher level overview of the structure, parts and components of the application.

R&D Doc

The R&D Doc section, contains the notes used to keep a record of how problem domain issue have been addressed, which options have been tried and considered and what is the current implementation strength and weaknesses.

  • Component/part description

  • Related projects. Eg parts that look good, or previous implementations. But have been used for current implementation options.

  • Implementations Options considered

  • Current implementation

  • What needs refactoring

The Stack

Trello board

Gitbooks

And/or, and in particular the .

For .

There's also a quick 10 min presentation at '17 NYC (covers version 1.0.7)

Not only node coupled with npm and yarn package management system allows for a modular component based approach but also projects such as (formerly node-web-kit) and , allow to use web technology to build cross platform desktop applications that with minimal code change can be ported to an equivalent web app version.

Most recently, running on chromium enabled close integration of web technologies with video editing software such as Premiere. This makes is possible to create "panels" that have access to the functionality of the video editing software, opening up the possibility to more closely integrate with the video editor and producer's workflow.

Traditionally video and audio have been like black boxes on the web, often having to recur to flash to provide video capabilities to pages. With the introduction of the HTML5 video tag, Javascript libraries like videos make it easier to manipulate the video, treating it as a Javascript object. However this turns the video into "a ‘black box’ we can do something with", such as triggering events at defined timecodes. It does not allow us to directly obtain the content of the video in a programmatic way and to manipulate the result. An example would be to take from the video the content of the quote and then analyse this to identify keywords and key topics. Another example would be to search what has been said in the video, find the quote and trim the video segment accordingly. For more of

As explored in a previous project, , I believe it is by combining video with its corresponding time-coded transcription that we can provide a direct programmatic solution.

After some research, it turned out that the best output for the "digital paper-editing"(a selection of text from different transcriptions) in autoEdit export part, as a "data interchange format" that was able to connect to a video editing software is an . Which stands for Edit decision list, a plain text file format, that describes a video sequences and is widely compatible with non linear video editing software.

In subsequent version it turned out that packaging the app as a Adobe Panel () can make the creation of a sequence in Adobe Premiere even more frictionless. As opposed to when importing an EDL the clips are offline and need to be reconnected.

Check out the for more on this.

Then it also contains a , and a to serve as a checklist before every deployment, as well as a series of appendix to collect technical info and implementation details relevant to the project , such as for example the , , , to mention a few.

The part is structured a round the 5 higher level parts that make up the app. Each one following this :

If you are not familiar with , , or not sure were to start to get an overview to familiarise yourself with this project, check out the to get an overview of the stack and see the minimum you need to know to get up to speed with this project.

Each release has it's own trello baord that gets duplicated/cloned to move on for subsequent releases. which will be kept for record, and there is .

Trello board is used for planning and project management. (Altho at the moment needs cleaning up and it's mostly gathering notes for feature ideas) are used for bug handling. There is also a that provides a trello like view of the github issues to help with organisation.

This documentation is written using gitbooks, and synced with a for convenience and backup.

gitbook version of the talk
section on digital paper-editing
an overview of how this has been implemented in autoEdit, from a user's perspecitve, check out the user manual
textAV
NWJS
Electron
Adobe Common Extensibility Platform (CEP)
a discussion of relevant projects that tackle this problem check out this article
quickQuote
EDL
Adobe Common Extensibility Platform (CEP)
roadmap section
QA section
db setup
prerequisite
ffmpeg & ffprobe packaging in electron
template structure
Node
Electron
backbone
prerequisite section
For instance this the trello board for v1.0.6
one to keep track for the next release v1.0.13
Github issues
waffle dashboard
github repo
autoEdit2
For more on paper-editing and how this is used to craft compelling stories out of video interviews, check out the slides from this workshop
http://pietropassarelli.com/wip_london_july2016.htmlpietropassarelli.com
LogoIntroductionHow to tell compelling stories out of video interv
LogoIntroductionautoEdit 2 User Manual
LogoautoEdittextav-event-2017
https://trello.com/b/Md913hIAtrello.com
High level overview of the parts