Motion Interpolation for Glitch Aesthetics using FFmpeg part 0

As you may have seen in this blog post I made use of FFmpeg’s minterpolate motion interpolation options to make all of the faces morph. There’s quite a few options for minterpolate and many different combinations of options that can be used. i had to consult Wikipedia to figure out exactly what the different motion estimation algorithms were but even with that information I couldn’t visualise how it would change the output. To add to this how I’m using minterpolate isn’t a typical use case.

To make things easier for those wishing to use FFmpeg’s minterpolate to create glitch aesthetics I have compiled 36 videos each showing a different combination of processing options. The source video can be seen below and features two of my favourite things: cats (obtained from here) and rainbows.

I’ve slowed it down so that you can see exactly what’s in the video, but the original can be downloaded here.

The base script used for each video is:

ffmpeg -i cat_rainbow_original.mp4 -filter:v "setpts=62.5*PTS,minterpolate='fps=25:mb_size=16:search_param=400:vsbmc=0:scd=none:

In part two of March’s Development Update I explained why I set scd to none and search_param to 400. I could have of course documented what happens when all of the processing options are changed but that would result in me having to make hundreds of videos! The options that were changed were the mc_mode (motion compensation mode), me_mode (motion estimation mode), and me (motion estimation algorithm).

Test conditions

These videos were created using FFmpeg 7:4.1.4-1build2, installed from the Ubuntu repositories, on a Dell XPS 15 (2017 edition) with 16GB memory, a i7 processor and an Nvidia GeForce GTK 1050 graphics card, all running on Ubuntu 19.10 using proprietary drivers.

I don’t have a Windows or Mac machine, and haven’t used other Linux distributions so can’t test these scripts in those conditions. If there’s any problems with getting FFmpeg on your machine it’s best that you contact the developers for assistance.

Observations

My first observation is that the esa me_mode takes frikkin ages to complete! Each video using this me_mode took about four hours to process. I did consider killing the script but for completeness I let it run.

Using bilat me_mode produces the most chaotic results by far. Just compare 026_mc_mode=obmc_me_mode=bilat_me=epzs.mp4 to 008_mc_mode=obmc_me_mode=bidir_me=epzs.mp4 and you’ll see what I mean.

For a video of this length nearly all of the scripts (except for those using esa) took between 30 seconds and 1 minute to complete, and that’s on machines with and without a GPU. This is good news if you don’t want to have to carry around a powerhouse laptop all the time.

All of this reminds me a bit of datamoshing. It’s more predictable and controllable, but the noise and melty movement it creates, especially some of the ones using bilat me_mode, remind me of the bloom effect in datamoshing. This could be down to the source material, and I’d be interested to see experiments involving datamoshed videos.

Let’s a go!

With that all said let’s jump into sharing the results. As there’s 36 videos I’ll be splitting it over nine blog posts over nine days, with the last being posted on 28th March 2020. Each will contain the script I used as well as the output video. Links to each part can be found below:

Glitch Art Workshops

Glitch art is a form of art that is created by introducing digital or analog errors to media.

Join MakoEducation and Near Now Fellow Antonio Roberts in this glitch art workshop which teaches participants what glitch art is and how to create their own artwork.

Attendees will learn how to create glitch art images, videos and GIFs. We’ll be encouraging artists of the future to experiment with new tools and techniques.

Attendees learn a range of digital skills in the areas of photography, image editing and video production.

No previous experience is necessary to take part in this workshop.

PLEASE NOTE THIS EVENT IS AIMED AT CHILDREN AGED 8-13. ADULTS PERMITTED ONLY TO ACCOMPANY CHILDREN. ALL CHILDREN MUST BE ACCOMPANIED BY AN ADULT.

Potentially Harmful

At the beginning of January I was approached by MTV to make a new visual ident for them. If you haven’t seen, since 2015 they have been approaching a wide range of artists to make these idents. They even received two D&AD awards for them. Previous ones that I’d seen included ones by Eva Papamargariti and Johnny Woods. It was a really big surprise to me and I was more than flattered to be included amongst the ranks of these great artists. Regarding the ident, they were really open to how I approached this. The usual caveats of having no nudity or violence applied, but otherwise they just wanted their logo at the end of a 15 second animation.

The finished piece, shown above, is essentially a glitchy world of shape and colour. This video was shown internationally on MTV except in the UK and the US.

Originally I had finished the ident at the beginning of February, with an air date set for beginning of March. However, on the 1st March we all received some surprising and bad news. Ofcom, who regulates TV and radio in the UK, had some issues with the video. Using their automated software they detected that at a few points in the animation it contained potentially harmful flashes. The full text of the error:

Click for full error description

Click for full error description

As per ITU-BT-1702 and Ofcom guidelines, Video Sequence starting at 01:00:14:333 may contain ‘Potentially Harmful Flashes(Luma Flash)’ for a duration of 3.467 seconds

I had no idea what this meant, but looking back on the video I could see that there was a lot of flashing. A clip from the animation illustrates this:

To further indulge my curiosity I decided to look up exactly what ITU-BT-1702 is. The resulting PDF, Guidance for the reduction of photosensitive epileptic seizures caused by television, gives a very detailed breakdown of research into photosensitive epilepsy and how images being broadcast can affect this:

Extensive studies on the subject of photosensitive epilepsy, which have taken place around the world, have led to formulation of this Recommendation. The guidance proposed in this Recommendation is for the protection of the vulnerable section of the viewing population who have photosensitive epilepsy, and who are therefore prone to seizures triggered by flashing lights, including certain types of flashing television images. Broadcasting organizations are encouraged to raise awareness among programme producers of the risks of creating television image content which may induce photosensitive epileptic seizures in susceptible viewers. Appendices 1 to 5 provide additional information on this subject.

Having been a teenager at the time that episode of Pokemon was causing troubles, I was aware of the need to reduce flashing images on TV but I definitely wasn’t aware the extent to which there are guidelines and studies supporting this. Even if you have no intention of having your work being broadcast on (UK) television it is definitely worth reading the document and checking out some statistics and pretty graphs!:

mtv_graph

With this new knowledge I edited the video, submitted it for approval and then sat glued to my TV in anticipation. However, the edited video had the same errors at different locations and was subsequently rejected again. And so I edited. And it was rejected. And so I edited. And it was rejected. And so I edited. And it was rejected. And so I edited. And it was rejected. After four edits I finally had dealt with issue ITU-BT-1702 but then on the fifth edit it was rejected yet again but with a new error:

Click for full error message

Click for full error message

As per Ofcom guidelines, Video Sequence starting at 00:00:11:300 seems to contain ‘Potentially Harmful Patterns’ for a duration of 533 millisecs.

I consulted the great Google again for information on these guidelines and found this document Annex 1 of the document specifies:

Ofcom Guidance Note on Flashing Images and Regular Patterns in Television

  • 5. A potentially harmful regular pattern contains clearly discernible stripes when there are more than five light-dark pairs of stripes in any orientation. The stripes may be parallel or radial, curved or straight, and may be formed by rows of repetitive elements such as polka dots. If the stripes change direction, oscillate, flash or reverse in contrast they are more likely to be harmful than if they are stationary. If the patterns obviously flow smoothly across, into, or out of the screen in one direction they are exempt from restriction.
  • 5.1. Potentially harmful patterns are not permitted when either of the following two conditions apply:
  • i. the stripes are stationary and the pattern occupies more than 40% of the displayed screen area; or
  • ii. the stripes change direction, oscillate, flash, or reverse in contrast and the pattern occupies more than twenty five per cent of screen area; and in addition to either of the above two conditions applying, when
  • iii. the screen luminance of the darker bars in the pattern is below 160 cd.m-2 and differs from the lighter bars by 20 cd.m-2 or more (see notes 1 and 2).

mtv_stripes_1

mtv_stripes

mtv_stripes_2

The section of the animation specified was towards the beginning, but in general the animation makes heavy use of striped patterns. This left me with a really tough decision. Removing the stripes completely could seriously and negatively affect the piece. It was at this stage that I began to investigate. Although the rejection of the edits was annoying, what was more frustrating was the waiting period between submitting an edit and it being assessed and eventually rejected. If I could have done the verification myself at home this would have been solved! So, I looked into software.

Ofcom appears to be using Baton for its quality control. No information on price is immediately available on the website but you bet it’s proprietary! Similar quality control software from the likes of Vidcheck are similarly proprietary. So, is there open source other software available for quality control, or at least checking for flashes or potentially harmful patterns?

No.

The closest open source software that does quality control is QCTools by Bay Area Video Coalition but, as far as I can tell, it cannot check for flashes or harmful patterns. Any developers writing open source software tackle this task would I’m sure be greatly welcomed!

In the end MTV made the very difficult decision to air the ident everywhere except for the UK. Although unfortunate it is understandable as causing injury isn’t something either of us wants to do! It was still an incredibly interesting learning exp‪erience for me and got me thinking about how my work, glitch aesthetics, and artwork by others with a similar aesthetic can actually be broadcast on television.

Although I have moved on from utilising only glitched visuals my work still retains many of its influences and aesthetics. By its nature these visuals often appear bright, colourful, glitchy and will probably have lots of flashing imagery. To me this raises the question: In order for glitch art to be broadcast on TV and in film do the glitches have to be removed, or at least tamed in some way? Exactly where is the line drawn between noisy and potentially harmful. Of course, having easy access to software would help make drawing these lines easier and, should I ever have acceses to it, I’d be very interested in throwing various glitches at it to analyse the results. How much of my existing artwork would pass or fail these tests?

Until these questions can be address I’m just grateful for the chance to challenge myself and MTV and push the boundaries, if only a little bit, on what glitches are acceptable and which are potentially harmful.

Maker Monday, 29th February

On 29th February I’ll be leading the workshop at Maker Monday at Birmingham Open Media.

makermondayglitch

Glitch Art describes the process of misusing and reappropriating
hardware and software to make visual art. In this part theory, part practice workshop we will consider ways in commonly used programs can be misused to interpret various data in different ways.

For the practical part of the workshop you’ll need a laptop/computer (Mac/Windows/Linux) with the following software installed:

Tickets for the event are free, so get in on it now! There’ll be pizza as well 🙂

An Introduction to Glitch Art

In this part theory, part practice workshop participants will be given a brief introduction into Glitch Art and its impact on art, music and popular culture. Participants will then have a practice session where they will delve into the common techniques utilised by glitch artists. Participants will need a laptop/computer (Mac/Windows/Linux) with the following software installed: GIMP, Audacity, a hex editor, and a good text editor such as Notepad++.

Bus Tops video

If you weren’t able to make it out to the streets of London in February to catch the Bus Tops here’s a video featuring a selection of the works:

You can see one of my pieces show up at 2:08. There’s some more videos from the project on al4ie’s YouTube channel