Development Update – December 2019 part 3

In this final part of this three-part Development Update I’ll be going over installing Xuan Ye‘s work in the Bcc exhibition. This work posed a similar challenge to Scott Benesiinaabandan’s work. I needed to automatically load a web page except this time I needed to allow for user interaction via the mouse and keyboard.

The artwork isn’t online so I’ll again go over the basic premise. A web page is loaded that features a tiled graphic with faded captcha text on top of it. The user is asked to input the text and upon doing so is presented with a new tiled background image and new captcha. This process is repeated until the user decides to stop.

Bcc:

I could have installed this artwork on a Raspberry Pi but thankfully I had access to a spare Leneovo ThinkPad T420 laptop, which negated the need for me to buy a keyboard and screen (#win). The laptop is a refurbished model from 2011 and was running Windows 7 when I got it. It is possibly powerful enough to handle a full installation of Ubuntu but I didn’t want to risk it running slowly so instead I installed Lubuntu, which is basically a lightweight version of Ubuntu.

As I had installed Scott’s work I already knew how to automate the loading of a webpage and how to reopen it should it be closed. The main problem was how to restrict the user and keep the user from deviating from the artwork. Figuring this out became a cat and mouse game and was never 100% solved.

Whilst in kiosk mode in Chromium pretty much all of the keyboard shortcuts can be used. This means that a moderately tech-savvy user could press Ctrl + T to open a new tab, Ctrl + O to open a file, Ctrl + W close the browser tab, Alt + F4/Ctrl + Q to quit the browser or basically any other shortcut to deviate from the artwork. Not ideal!

Bcc:

My first thought was to try and disable these shortcuts within Chromimum. As far as I could tell at the time there wasn’t any option to change keyboard shortcuts. There must be usability or security reasons for this but in this situation it sucks. After a bit of searching I found the Shortkeys extension which allows for remapping of commands from a nice gui 🙂 Only one problem. I tried to remap/disable the Ctrl + T command and got this error.


More information here.

Drats! I tried its suggestion and it still didn’t work. Double drats! Eventually I realised that even if did disable some Chromium-specific shortcuts there were still system-wide ones which would still work. Depending on your operating system Ctrl + Q/W will always close a window or quit a program, as will Alt + F4, Super/Windows + D will show the desktop, and Super/Windows + E/Shift + E will open the Home folder. I needed to disable these system-wide.

LXQT has a gui for editing keyboard shortcuts. Whilst it doesn’t allow for completely removing a shortcut, it does allow a user to remap them.

As you can see from the screenshot above I “disabled” some common shortcuts by making them execute, well, nothing! Actually it runs “;”, but still that has the effect of disabling it. Huzzah! But what about the other keyboard shortcuts, I hear you ask. Well, this is where I rely on the ignorance of the users. Y’see, as much as it is used within Android phones and basically most web servers, Linux/Ubuntu is still used by a relatively small amount of people. Even smaller is the amount of people using Lubuntu or another LXQT-based Linux distribution. And even smaller is the amount that work in the arts, in Birmingham, and would be at Vivid Projects during three weeks in September, and knew how I installed the work, and… I think you get my point.

During the exhibition anyone could have pressed Ctrl + Shift + T to open a terminal, run killall bcc.sh to kill the script that reopens Chromium, undo the shortcut remappings and then played Minecraft. I was just counting on the fact that few would know how to and few would have a reason to. After all there was some really great art on the screens!

After the exhibition was installed Jessica Rose suggested that one simple solution would have been to disable the Ctrl key. It’s extreme but technically it would have worked at stopping users from getting up to mischief. It would have had the negative effect of preventing me, an administrator, from using the computer to, for example, fix any errors. The solution I implemented, whilst not bullet proof, worked.

That’s the end of December’s Development Updates. Installing Bcc was frustrating at times but did push me to think more about how people interact with technology in a gallery installation setting. It’s never just a case of buying expensive hardware and putting it in front of people. There needs to be processes – either hardware or software based – that protect the public and the artwork. It doesn’t help when lots of technology is built to be experienced/used by one user at a time (it’s called a PC (personal computer) for a reason y’all). Change is no doubt to make it more about groups and collaboration but, y’know, it’ll take time.

Development Update – December 2019 part 2

The next artwork that was challenging to install was Monuments: Psychic Landscapes by Scott Benesiinaabandan.

Bcc:

I won’t be showing the full artwork as all of the artworks were exclusive to Bcc: and it’s up to the artists whether they show it or not. On a visual level the basic premise of the artwork is that the viewer visits a web page which loads an artwork in the form of a Processing sketch. There is a statue in the centre which becomes obscured by lots of abstract shapes over time whilst an ambient soundtrack plays in the background. At whatever point the viewer chooses they can refresh the screen to clear all of the shapes, once again revealing the statue.

On a technical level the artwork isn’t actually that difficult to install. All that needs doing is opening the web page. The difficult part is controlling user interaction.

If you’ve ever been to an exhibition with digital screen-based artworks which allow user interaction via a mouse, keyboard or even touch screen then you’ve probably seen those same screens not functioning as intended. People always find a way to exist the installation and reveal the desktop or, worse yet, launch a different program or website. So, the choice was made very early on to automate the user interaction in this artwork. After all, aside from loading the artwork, the only user interaction needed was to press F5 to refresh the page. How hard could it be?

Well, it’s very hard to do. Displaying the artwork required two main steps:

  • Launch the web page
  • Refresh the artwork after x seconds

Launch a web page

Launching a specific web page on startup is a relatively easy task. Raspbian by default comes bundled with Chromium so I decided to use this browser (more on that later). The Chromium Man Page says that in order to launch a webpage you just need to run chromium-browser http://example.com. Simple! There’s lots of ways to run a command automatically once a Raspberry Pi is turned on but I settled on this answer and placed a script on the Desktop, made it executable (chmod +x script.sh), and in ~/.config/lxsession/LXDE-pi/autostart I added the line @sh /home/pi/Desktop/script_1.sh. At this stage the script simply was:

#!/bin/bash

while true ; do chromium-browse --noerrdialogs --kiosk --app=http://example.com ; done

I’ll break it down in reverse order. --kiosk launches the browser but in full screen and without the address bar and other decorations. A user can still open/close tabs but since there’s no keyboard interaction this doesn’t matter. --noerrdialogs prevents error dialogs from appearing. In my case the one that kept appearing was the Restore Pages dialog that appears if you don’t shut down Chrome properly. Useful in many cases, but since there’s no keyboard I don’t want this appearing.

I wrapped all of this in a while true loop to safeguard against mischievous people who somehow manage to hack their way into the Raspberry Pi (ssh was disabled), or if Chromium shuts down for some reason. It’s basically checking to see if Chromium is open and if it isn’t it launches it. This will become very important for the next step

Refresh a web page

This is surprisingly difficult to achieve! As mentioned before, this piece requires a user to refresh the page at whatever point they desire. As we were automating this we decided that we wanted a refresh every five minutes.

Unfortunately Chromium doesn’t have any options for automatic refreshing of a web page. There are lots of free plugins that offer automatic refreshing. However, at the time that I tried them they all need to be manually activated. I couldn’t just set it and forget it. It could be argued that asking a gallery assistant to press on a button to activate the auto refreshing isn’t too taxing a task. However, automating ensures that it will always definitely be done.

At this point I looked at other browsers. Midori is lightweight enough to be installed on a Raspberry Pi. It has options to launch a web page from the command line and, according to this Stackexchange answer it has had the option since at least 2014 to refresh a web page using the -i or --inactivity-reset= option. However, I tried this and it just wasn’t working. I don’t know why and couldn’t find any bug reports about it.

It was at this point that I unleashed the most inelegant, hacky, don’t-judge-me-on-my-code-judge-me-on-my-results, horrible solution ever. What if instead of refreshing the browser tab I refreshed the browser itself i.e. close and reopen the browser? I already had a while true loop to reopen it if it closed so all I needed was another command or script that would send the killall command to Chromium after a specific amount of time (five minutes). I created another script with this as its contents:

#!/bin/bash

while true ; do sleep 300 ; killall chromium-browser ; done

The sleep command makes the script wait 300 seconds (five minutes) before proceeding onto the next part, which is to kill (close) chromimum-browser. And, by wrapping it in a while-true loop it’ll do this until the end of eternity the exhibition. Since implementing this I noticed a similar answer on the Stackoverflow site which puts both commands in a single file.

And there you have it. To refresh a web page I basically have to kill it every 300 seconds. More violent than it needs to be!

Development Update – December 2019 part 1

I took a bit of a break from writing the Development Updates. September was pretty busy with Bcc: (more on that below) and then I was completing a commission for Will’s Kitchen/The Shakespeare Birthplace Trust and preparing for my solo exhibition, We Are Your Friends.

With all of that now completed I’m writing a few posts about one project in particular: Bcc:

The Bcc: exhibition opened at Vivid Projects on Friday 6th September. It was a collaboration between Vancouver-based Decoy Magazine and Birmingham-based Vivid Projects. The exhibition featured a curated selection of works from Decoy Magazine’s online art subscription service called Bcc:. The basic premise is that each month you’d get specially commissioned art in your e-mail inbox.

Bcc:

Bcc:

After being part of Bcc: in 2018 I suggested to Lauren Marsden, the Curator and Editor of Decoy Magazine, that it could possibly become an IRL exhibition at Vivid Projects. At the time I was still working there so I worked on getting most things in place to get the exhibition going. Then I left in 2019. Because of my prior involvement in Bcc: and the massive technical challenge involved in installing the work (more on that later) I was asked to produce the exhibition.

Depending on how you look at it the technical aspect of installing the exhibition could be very simple. Most of the works in Bcc: were short movies and animations/gifs, and Vivid Projects has long used the Adafruit Raspberry Pi Video Looper to handle playing videos.

Some works, however, required more attention. There were some works that were interactive websites, some that were animated gifs and some that require additional hardware. Prior to the exhibition this probably didn’t present any problems as the works were viewed by most likely one person on their personal phone or computer. The challenge comes when it’s on a shared computer in a public environment. Additionally, operating the works needs to be as hands off as possible. That is, I didnt want it to be the case that myself or another technician had to be on hand every day to go through complicated procedures to turn on all of the work. They needed to be automatic. With 17 works each needing their own computer/Raspberry Pi there was a lot to prepare. Over the next few posts I’ll take you through some of the works and their technical challenges:

Playing gifs on a raspberry pi

Of the 17 works on show in the exhibition 10 were animated gifs. To stay true to the small nature of animated gifs (don’t get me started on the concept of HD gifs) we decided to display the gifs on the Official Raspberry Pi 7″ Touchscreen Display. This proved to be a really good decision overall. It required that visitors get really close to the works and spend time with a format that can sometimes be a bit throwaway.

Bcc:

As mentioned before, for a long time Vivid Projects has used the Adafruiit Raspberry Pi Video Looper software to play videos. It works (mostly) great with the exception that it doesn’t play animated gifs. The main underlying software, omxplayer, only supports video files. Even the supplied alternative player, hello_video, also only plays video files.

Your immediate though might be to just convert the animated gifs to video files. Whilst this “works” there is always the danger that in converting a file you reduce the quality of it. For an artist like Nicolas Sassoon, who makes pixel-perfect animations that match a specific screen size, this would be unacceptable. So I went on a journey to find a way to play gifs.

The requirements for the software is that it should operate in a similar way to the Adafruit software and play a gif on loop with little or no pause between loops. It should play in the frame buffer (i.e. without needing to load the desktop) and it should make use of the GPU (helps prevent screen tearing). And for a bonus it should be able to play a series of gifs one after the other. Simple, right?

TL;DR: There isn’t a reliable way, I had to convert to a video.

Some of the solutions I saw were saying to use Imagemagick to play the gifs. This wouldn’t work as I would need to launch the desktop. Then, I’d need to script it to go full screen, centre the gif, change the background to black etc.

FBI and FIM don’t support animated gifs, although they are useful if you ever want to play a slideshow of static images.

feh is another image viewer that uses the framebuffer. However, it also doesn’t support animated gifs and, according to this response from the author, this is by design.

This suggested solution of converting to images kinda works but doesn’t take into account if each animation frame has different durations (see this GIMP tutorial for example on how to use it). With that in mind for this to work I would need to get the duration of each frame in each of the 10 gifs, separate the gifs into their individual frames, and then tell feh to play each frame for it’s specified duration. So, this method could work but it would require a lot of work!

This thread on the Raspberry Pi forum did provide a possible solution which I didn’t try but it also pointed me to FBpyGIF, which was certainly the most promising of the solutions. However, a couple of problems prevent me from using it. Still very promising though!

Finally, I tried one of the various GIF Frames that play a folder of animated gifs on loop. Sounds like it works but there’s screen tearing on some fast-moving gifs. I’m guessing this is because it doesn’t have hardware acceleration and/or because it uses Chromium to play the gifs.

Soooooo after all of this I felt a bit defeated and I decided to just convert the animated gifs to videos. I used Handbrake and noticed no loss of quality in the conversion. Even if there was, on a 7-inch screen it’d be quite hard to see. Using the Adafruit player/omxplayer I was initially having some issues with aspect ratio. Even with –aspect-mode set to fill stretch or letterbox, the videos were being stretched to fill the screen. To illustrate take the following video, which is 1024×68/4:3.


(fyi it was made using Natron and this script to add in a timecode)

When play on the screen it is stretched to fill the screen.

The Raspberry Pi touch screen has a resolution of 800 x 480, which is a 5:3 aspect ratio. Most of the videos and animated gifs were HD/16:9 so would be letterboxed by default.

So I had the bright idea of padding each video so that it was exactly 800×480.

Now, the Adafruit player/omxplayer says it can play any video which is H.264 encoded but I’ve had some troubles in the past, so whenever I’m given a video I usually convert it using Handbrake with the Fast 1080p30 preset. These settings have always worked for me but for some reason on this occasion the video was stuttering a lot! What was strange was that the original videos (the animated gifs converted to videos without resizing) played fine. Even after they were run through Handbrake. Why when they were converted to 800×480 size did they stutter?

It was two days before the exhibition opening that I remembered that some time in 2016 I had an issue with omxplayer in that it didn’t play videos if the video didn’t have an audio track. Why? I don’t know. Maybe audio was the problem in this scenario too? It was worth a try and so I decided to disbale the audio track using the -n -1 option. This doesn’t just turn the audio down, it disable encoding of it. And guess what. IT WORKED!

I have absolutely no idea why this worked or why the error ocurred in the first place. Here’s the extra arguments that I included on line 107 of video_looper.ini.

extra_args = --no-osd --audio_fifo 0.01 --video_fifo 0.01 -n -1 --aspect-mode stretch

All of that just to play animated gifs! Now that I had the code perfected copying it to all of the other Raspberry Pi’s was simple. If the aforementioned softwares had animated gif playback by default then this would’ve been solved much quicker but for now it seems the most reliable way to play animated gifs on a loop on a Raspberry Pi is to convert them to video.

Development Update – August 2019

What’s happening on Twitter

The following is compiled from a bunch of Tweets that I made in December 2018. After reading you’ll see why I have to write it here! While it is not directly related with programming or making art, it does help with Getting Things Done, so I decided to include it here.

Like many people I’ve started to remove myself from a lot of social media websites. First was Facebook in 2017. The reason for this is that was really annoyed that it was using nostalgia to manipulate me into staying on the website. In shoving 10 year-old photos into my view through the On This Day feature it was giving me little hits of dopamine by reminding me of the good ol’ times, even if they were 10 years ago with people that, for whatever reason, are no longer part of my life.

One solution to this was to make sure that Facebook only had recent information about me. I started manually deleting anything that was more than 2 year old. I eventually found a Chrome plugin (use at your own risk) that made it easier to do but this process was a chore that ultimately didn’t solve the fact that Facebook was the problem. After about a year I left unannounced. After deleting my account, of course.

My “relationship” with Twitter is a bit different. I’ve always preferred it over Facebook as it isn’t as intrusive, at least not directly. It doesn’t constantly ask you to share who you’re dating, identify your family, upload photos from your night out or tag your friends in everything. Instead it felt like it was more concerned with what was happening at that moment.

Like Facebook, though, I became a bit concerned with how much data about me it was storing. I started using the website in 2008 (Facebook in 2007) and have used it almost daily since then. Over that time I have grown and changed as a person many times over. I don’t want this history to be fully documented and, more importantly, available for anyone to browse through. Whilst the majority of the 40k tweets I accumulated over that period probably consists mostly of cat gifs, memes and the word “lol”, maybe there’s there events that I’d rather not have documented, like Tweets showing friendships and relationships falling apart, embarrassing photos of myself or others on nights out, or even just me saying something that was totally out of order.

I’m glad that I have friends (and enemies) that have called me out on my bullshit and hope that they continue to point out times when I do something wrong. However, I’d rather that the trail of data I leave on these sites that I use every day reflected me as I am now, not who I was 10 or even 20 years ago.

So, I went on a mission to find a way to keep my Tweets current. I needed a tool, or tools, that would automatically delete Tweets older than a certain time period.

A lot has been written about Tweetdelete. However, I don’t want to rely on a third party service. Many people do trust the service, but there’s always risks in using third party services, especially when they have access to a lot of your information. Then there’s the risk that it could one day shut down so I decided that I wanted something that I could deploy myself.

Deploying your own script requires that you register a developer account on Twitter.

Delete tweets is a Python script that let’s you delete tweets and specify a cut off date. However, to run it you need to download your Twitter archive. At the time of writing this can only be done once a month and has to be done manually. So, you could automate the running of the script but there’s still manual intervention required.

This Python script is similar but it lets you specify cutoff date as a number of days, not dates. Still, it requires downloading your Twitter archive manually.

This Ruby script works perfectly! You specify cutoff point in days and then when it is run it deletes any tweets older than that cutoff point. It even has the option to put in the ID of Tweets that you want to save. It only requires a developer account and you don’t need to download your archive.

There’s even a companion script that removes Likes. This doesn’t have any options for date cutoff but in my case it doesn’t matter. Once I’ve liked something once it doesn’t mean that I like it (or anything else that person has posted) forever so I’m not sure why I need to have my likes recorded and archived.

I decided to install both scripts on an always-on Raspberry Pi. Installing them took a bit of time due to it needing to install a bunch of Ruby gems. Once it was installed I set up a cron job to run the script at regular intervals. I have mine set to twice a day and to only keep the last two weeks of tweets. I feel that that is enough time for the tweets/memes to have whatever impact that they’re going to have. After two weeks they’re gone.

All of this effort to manage my experience of using Twitter might not be a solution and instead might be more of a distraction from the fact that the problem is Twitter, and maybe even social media in general. There have been many efforts from individuals to make social media better. On Facebook there is F.B. Purity which helps remove things like adverts, the On This Day feature and other things.

One of my favourite tools that I still use is the Facebook and Twitter Demetricator from Ben Grosser. These desktop-only tools remove mentions of the number of Likes, replies and retweets a post gets so that you can focus on the cat memes important things. These plugins have been getting a lot of attention recently. See Ben’s Instagram for more.

This of course doesn’t solve social media’s problems but just makes my experience of it just that little bit less stressful.

Development Update – July 2019

Select objects of similar size in Inkscape

For the AlgoMech 2019 festival in June I created a new performative drawing piece, A Perfect Circle. The piece is about how we interface with computers that analyse our activities. It consists of a video and accompanying plotter drawings.

Making A Perfect Circle presented me with a few challenges. The make the video element I hacked together a couple of Processing scripts that did basic motion tracking by following a user-specified colour. It would draw these lines, creating new lines (instead of adding to an existing line) at each major turn and giving them a unique colour.

The next stage was to export those drawn lines to SVGs (or PDFs) so that I could export them to Inkscape and then a plotter. Fortunately Processing already has functions for exporting to SVGs. Unfortunately for me if I were to implement this as is suggested in the help file it would export both the drawn line and the background video as a still frame. I produced a very hacky workaround (with help from Ben Neal) which “works” but produces a few unwanted artefacts.

Before I go on I should probably explain what a plotter is as the unwanted artefacts relate to it. For this I will copy from the Wikipedia article on plotters:

The plotter is a computer printer for printing vector graphics. Plotters draw pictures on paper using a pen. In the past, plotters were used in applications such as computer-aided design, as they were able to produce line drawings much faster and of a higher quality than contemporary conventional printers, and small desktop plotters were often used for business graphics.

At home I have a Silhouette Cameo 2 vinyl cutter. When using this great Inkscape plugin I can bypass Silhouette’s proprietary software and send artwork directly to the cutter from Inkscape. Thanks to a pen holder adaptor I can replace the vinyl cutting blades with a pen and turn the vinyl cutter into a plotter 🙂

Back to the Processing sketch. The hacky code that I made produced the desired lines but also it had lots of additional single-node paths/dots at the start of each line.

Removing these wouldn’t be very easy. Using Edit > Select Same > Fill and Stroke or Fill Color or any of the other options wouldn’t work as it would also end up selecting the lines. I then had the bright idea to select objects based on their size. All of the dots had a dimension of 4.057×4.000px, so in theory there would be an option like Edit > Select Same > Size. However, this is not so.

After a discussion on the Inkscape forum I opened a feature request on the Inkscape bug tracker to select objects of similar size. One thing I added to this was the idea of a threshold. Using this you could select objects that were within n% of the size of the selected object. If you’ve ever used GIMP you would have seen a similar function in its fuzzy selection tool This could definitely be useful if you trace bitmaps and it produces a lot of speckles. I also added a mockup to show how it could be applied to other options in the Edit > Select Same menu options.

Anyway, at the moment this exists as a feature request. I think Inkscape is concentrating on delivering version 1.0 of the software so I don’t expect to see this implemented any time soon. As with anything in the land of open source, if you’ve got the skills to do this please contribute!

In the end I used fablabnbg’s Inkscape extension to chain all (or most) of the paths into one big path. This made selecting the dots easier as I could just hide the big path(s) once they were chained together.

After that it was a simple case of sending it to the plotter!

Development Update – June 2019

Making digital art is quite a lengthy process and even moreso if you’re using non standard processes or making your own software. For awhile I’ve wanted to write about my processes and how I’ve overcome the bugs and problems. In what will hopefully be a regular series of blog posts I’m going to give a bit of insight into this process. Let’s go!

Convert Object texture coordinates to UV in Blender

For Visually Similar I wanted to texture each 3D model using lots of images found on the internet. Rather than create one single material containing a texture with all of the found images I instead decided I would add a material for each image texture and, using their alpha channels, composite them over each other.

If you’ve ever had to position something accurately on a UV map you’ll know how much of a pain it can be. So fortunately, in the Texture Coordinate node you can use the Object outlet to another object (usually an empty) as the source of its coordinates. This uses the reference object’s local Z direction as its up direction.

So far,so good, except it did not yet work in Blender’s new EEVEE rendering engine. Yes, yes, I know EEVEE is still under development and shouldn’t be used in production etc. Still, after doing a bit of research it looks like this is going to be implemented.

So, I had a rather smrat idea as a workaround. Could I take the UV coordinates generated by the Object oulet whilst using Cycles and paste those into the UV texture options using a Mapping node? Short answer: no. To do this I would need some sort of viewer or analyser node that would show me the data being output from a node. So, I suggested this idea on the Right-Click Select ideas website. A healthy discussion followed and hopefully something will come of it.

In the end I had to resort to baking the texture and then applying that to the 3D model. In doing this I learnt that baking a UV texture on a complex model will take a lifetime, and so I had to do it on a decimated model and then put that on the original, complex model. This, of course, created some unwanted artefacts. *sadface*

Since I originally encountered this problem it has actually been addressed in a Blender update! However, it only works at render time but it’s progress! 🙂

The search for a GrabCut GUI

Another big part in creating the Visually Similar artwork was the image textures themselves. The idea for the piece is that the textures would be related in some way to the 3D model. I decided from the beginning that I wanted to have some control over this and so I gathered the images through keyword searches and reverse image searches.

But then I needed to cut out certain parts of them. I wanted it to look like a rough collage, as if the images were pages in a magazine that had been ripped out, leaving behind tears and occasionally ripping through the important bits.

For awhile on of my Twitter friends, _xs, has had a bot on their feed that generates random collages. I haven’t studied the source code extensively but I’m guessing it does a keyword search and makes a collage out of the returned images.

What I was really interested in was how the images were cut out. It’s as if a sort of automatic feature extraction was used but wasn’t very accurate and so it left behind jagged edges that were almost reminiscent of the kind of ripped magazine aesthetic that I mentioned earlier.

Through a conversation with them I learned that they used a combination of automated object detection (to select the region of interest) and GrabCut to perform this automatic foreground extraction. Grabcut has been part of OpenCV for quite some time. Give it a region of interest (ROI) and it will attempt to extract the foreground.

_xs used this via the command line and automated the whole process. I needed a bit more control over defining the region of interest and so I needed a GUI where I could use a bounding box to select this. This is where the long hunt began.

OpenCV has its own GrabCut GUI example but it has an annoying flaw.

To select the ROI it displays the source image at full size. Meaning that if your source image is 4000 pixels wide it won’t fit on your screen (unless you have a fancy pants 4K screen). Not ideal when trying to select an ROI. What I needed was a way to scale the window to fit on my screen but still process a full resolution image.

If you search Github you’ll see a number of people have created GUIs for GrabCut, possibly for study assignments. However, each has their own problems. Some won’t compile, some resize the input and some have been abandoned. According to this 2006 article there was even once a GUI for GrabCut in GIMP. However, despite my best efforts I can’t seem to find it.

One night at OpenCode I learnt that OpenCV has a method for selecting an ROI! It even auto resizes the window but not the input image. Yay! So, I hacked it together with GrabCut and released my own very hacky Grabcut GUI. It appends to the file name the coordinates and dimensions of the ROI should you want to run this again but via the command line.

All this done with a mere seven days until the artwork had to be finished!

Typewriter text

For the Algorave at British Library in April I was asked to make a promotional video for it, which proved a difficult but for a very specific reason. I wanted to emphasise the liveness of live coding and show code being typed. For this I used the code supplied with Alex McLean aka Yaxu’s excellent Peak Cuts EP.

The effect of having the text appear word-by-word or letter-by-letter is often called the typewriter text effect. I’ve previously written about how to do this in Pure Data/GEM. I needed to have a bit more control than what I got in PD, and I needed to export as transparent pngs so this solution wouldn’t work.

Kdenlive once had such an effect built into its title editor. Other solutions that used Kdenlive use a mask to reveal the text, which produced more of a fading in effect that wasn’t ideal. It was also a lot of manual work! I had several hundred lines of text so doing this was going to add a lot of time.

Natron was the next contender. Since 2017 it has had a plugin for doing typewriter text but it’s a bit broken. In theory in gives me the most flexibility in how I create it but in practice I still can’t get it to render!

I also considered using ImageMagick and was even provided with a solution (that was written for Windows). As much as I like automation and command line software, for this very visual task I needed to see what I was working on.

Finally, I turned to Blender, which gave me a few options, including rendering the text as 3D objects within the Blender project itself. After failing to get this Blender addon to work I tried using Animation Nodes. Following a tutorial I was able to set up quite a typewriter effect quite quickly. However, this is where I encountered a bug. After around 10 frames of the text were rendered the rest of the frames would take forever to render. Even in EEVEE each frame was taking about 10 minutes to render. I have no idea why this was. Perhaps it’s because 2.8 is in beta. Maybe because Animations Nodes for 2.8 is also in beta. Beta beta beta. Either way it wasn’t working.

So I thought maybe I could “bake” the animation which would remove the Animation Nodes dependency and maybe speed up the render. Sadly this was also not to be. Text objects can’t be baked 🙁

In the end I had to do an OpenGL render of the animation to pngs with a transparent background. How this differs from a normal render is that it renders the viewport as is. So if you have your gizmos on there it’ll render them out as well. Not ideal but it worked.

I would like to think it all stopped there but it did not.

Blender can have a video or series of images be a texture. However, at the time this was not possible in 2.8 using EEVEE. To my joy, however, this was implemented only a couple of days after i needed it!

So that is some insight into how I make some of my art. There’s a lot of problem solving, lots of showstopping bugs and lots of workarounds. Somewhere in that process art is made! I’m hoping to do these every month but we’ll see how that goes.

Five Days of Pure Data – Live Coding

In the years that I’ve been creating things in Pure Data I have amassed quite a collection of unfinished and messy patches. Over the last few days I have been releasing a few of these patches and techniques that I implement when programming in Pure Data.

Live Coding

The last tutorial isn’t that much about actually programming with Pure Data and is more about performing with it. More specifically, live coding. Live coding takes programming to a performative level. There’s been a lot of writing recently about Live Coding so take a look around. I’ve been programming live visuals for Pure Data since 2014 (a lot happened that year). It’s not without it’s problems, some of which I written about in the past.

One problem I’ve found is displaying your patch alongside your visuals. For live coding musicians this isn’t much of a problem as the output of the live patching is sent to the sound card, not necessarily the screen. For people live coding visuals in Pure Data the output (projector) would be needed for both the visuals output and the patch. If only there was a way to overlay the patches onto visuals (like in Fluxus, Cyril and Livecodelab)!

The most reliable cross-platform method to do this is to use Open Broadcaster Software (OBS). OBS is a great piece of software used by many gamers for livestreaming. A great feature of it is to be able to combine multiple media sources into one to then stream or send to a projector or video file.

To overlay your patch onto your visuals first create a GEM window with some graphics in it. In OBS, under Source click the + button and choose Window Capture. Give the new source a name. And then, under Window choose your GEM output window.

Next, create a new Window Capture source and give it a name (perhaps Pure Data patch). You may have noticed that your patch completely covers your GEM source. No worries! Right-click on your Pure Data patch source and click on Filters. In the next window click on the + button and choose Chroma Key. Give the new key a name (White).

In the next window, under Key Color Type choose Custom and then select the colour. You should choose white (#FFFFFF). You may need to change the settings under Similarity and Smoothness to get the right look.

If you’ve just tried this in Pure Data vanilla you’ll notice that your objects are also transparent. I’m using Purr Data (which you should too). In this version of Pure Data all of the boxes are slightly grey, resulting in them not being transparent when you apply the Chroma Key.

Imagine a scenario where you’re collaborating on an Algorave set with a musician and there’s only one projector. Both will want to project their code but there’s only one projector. What to do?! Luckily a combination of OBS and desktop sharing has you covered.

TidalCycles code from yaxu

If the musician can share their desktop over a network then you can add that in as a source in OBS and do the usual thing of applying a chroma key filter. However, this requires you both be on the same network, which isn’t always possible or efficient. However, if one of you has a recent Android phone you can get over this hurdle and have high quality desktop sharing quality.

On your phone create a wireless hotspot. You don’t need to use mobile data for this so feel free to turn it off.

Connect both machines to this newly created network.

On the remote (musicians) laptop, open Desktop Sharing options and enable the option “Allow other users to view your desktop”. You can choose to require a password and have a user confirm on each new access request.

On the host (visuals) machine open Remmina Remote Desktop Client.

In this new window click New. In the next window change the Protocol to VNC – Virtual Network Computing. In the Basic tap below click on the … button.

You should see the name of the remote computer listed. Click on one of them.

Finally, press Connect. When you try to connect to the remote machine you may get a prompt asking the user to confirm that the connection request is permitted.

Once the connection is approved you should see the remote machine on your computer!!! 🙂 And now you can go back to OBS and add that window as a source, apply a Chroma Key effect and then overlay it onto your visuals.

Sadly you can’t then use this source to apply as a texture to your objects in Pure Data, but it’s a great start to merging visuals and music live code. I first tried out combining my own code and visuals at the Chemical Algorave in Newcastle and it went really well. I also made use of OBS’s “Apply LUT” filter to change the colour of my whole video output.

MVI_3763

Chemical Algorave

I then tried incorporating a musicians code at the Algroave at BUMP in Kortrijk.

A post shared by Antonio Roberts (@hellocatfood) on

I’m hoping that this will help lead to more collaborations between Algorave musicians and visual artists 🙂

Five Days of Pure Data – Stop Motion

In the years that I’ve been creating things in Pure Data I have amassed quite a collection of unfinished and messy patches. Over the next few days I’ll be releasing a few of these patches and techniques that I implement when programming in Pure Data.

Stop Motion

It was at the Co-Position meeting of the Libre Graphics Research Unit in 2012 that I first encountered Toonloop. It’s creator, Alexandre Quessy, gave a live performance using lights, Lego pieces and other objects. I was really quite in awe of how stop-motion was used to create quite an enjoyable performance.

Of course my first instinct was to try and recreate this but in Pure Data. I wasn’t the first to try, and I in fact have some memory of seeing Toonloop’s creator himself trying to write something similar in Pure Data although I can’t find any sources…

My first instinct to recreate Toonloop was to use [pix_write] to write a series of images and then play those back using [pix_image]. The problem there is that there’s no easy way to read an arbitrary set of images from a directory.

In the end I learnt about [pix_buffer_write] which allowed me to story an image frame into a buffer which I can then call back using [pix_buffer_read]. So that’s the basic functionality sorted! When I went to Databit.me in 2013 Axel Debeul helped to improve the patch a lot. The improvements allowed me to save a video from each animation. You can find the most recent version of the patch below

That’s where the problems start to arise, some of which I haven’t solved yet. The videos are created via [pix_record]. When a frame is captured it is sent to [pix_record] and then recording is paused. However, when you look at the saved video it has a really weird frame rate and doesn’t play smoothly. Even when the frame rate is set explicitly it somehow doesn’t work.

Don’t ask me about the cat ears

Perhaps making a video out of the animations is something to be done in post processing rather than in Pure Data. [pix_record] has always been a very complicated object to work with so perhaps I need to investigate further and try to find the right configuration of all of its many options.

Five Days of Pure Data – Infinite Scrolling

In the years that I’ve been creating things in Pure Data I have amassed quite a collection of unfinished and messy patches. Over the next few days I’ll be releasing a few of these patches and techniques that I implement when programming in Pure Data.

Infinite Scrolling

For a performance at ChipFest 10 in 2014 I wanted to recreate the look of a scrolling video game but within Pure Data. Sure, I could have just used a console emulator or recorded a clip and used that but doing it all within Pure Data allows me to have some more flexibility with how I perform. Here’s what I made:

I eventually didn’t use it all but I did use the a few techniques I learnt for future performances.

They key to those visuals and for things like 2D side-scrolling computer games is to have infinite scrolling background. That is, to have a single image which tiles and repeats seamlessly as it moves across the screen. Not sure what I mean? check out these examples from TV history:

Borrowing a bit from this answer on the Game Development Stackexchange, first you create a single tile of size, say, 512×512. You scroll it across the screeen horizontally, and once it’s position gets to 512px (or -521px) it jumps back to it’s original position.

If the original tile is repeated a few times and then offset so that the borders of the tile can’t be seen then suddenly you have an infinite scrolling background! To repeat an object in Pure Data we use the [repeat] object. This object repeats any object and its translations. So, if a [cube] has a translation of 1 on the X axis and is [repeat]ed 4 times, then the first [cube] is offset by 1, the second [cube] is offset by 1 in relation to the first [cube] (or offset by 2 in total). The third cube is offset by 1 in relation to the second cube (or 3 in total), and so on.

In action this looks like this:

Success!

You can also do the same with the Y and Z axis. For the Z axis you just need to decide where the vanishing point is. Or you could always use fog 😉

The patch for this is really quite simple but I’ve made it available for download below:

Five Days of Pure Data – Randomise Text

In the years that I’ve been creating things in Pure Data I have amassed quite a collection of unfinished and messy patches. Over the next few days I’ll be releasing a few of these patches and techniques that I implement when programming in Pure Data.

Randomise Text

In the early 2010s I had quite an interest in zines. I had co-organised the Birmingham Zine Festival and was quite regularly reading this. As a result of this in 2011 I started collaborating with a friend, Rebecca Evans, on a collaborative zine. The concept is that we would interpret each others’ way of working using our regularly used tools. Rebecca specialises in textiles and has quite a skill at crocheting, amongst other things. I, on the other hand, can barely operate a sewing machine and feel much more at home using a soldering iron or computer.

For Rebecca this meant trying to create some sense of noise and randomness but using sewing. The results, even if only tests, looked quite awesome!

For my own take on this I wanted to continue the text based work that I was creating at the time. Rebecca two poems to work with, Lovesong by Ted Hughes and Lady Lazarus by Sylvia Plath.

I wanted to remix the works by jumbling the words and/or sentences. Looking at recent(ish) work you can see that this would become a bit of a theme in my work and references. spɛl ænd spik, Silver Screen Changeable, and my reinterpretation of Variations on a Themeby Casey & Finch by Erik Bünger each look at cutting up and rearranging words.

Cutting up words and rearranging them can be achieved in many languages using just a few lines of code. One method is to put each word or sentence into an array and then print those indexes in the array randomly. In Pure Data it isn’t as easy.

For most cases Pure Data has the [tabread] and [tabwrite] objects. To use [tabwrite] you specify the index and then write data to it. You then move onto the next index number and write more data to it. To read the data back you do the inverse but with [tabread]. This works great with numbers but not with text.

Y’see, Pure Data has several data types. Just like php, python and others have strings, numbers, text, etc, Pure Data has symbols, floats, integers etc. The character “6”, for example could be a number (default) or a symbol – using a combination of a number box, [floattosymbol] and then a symbol box. As a number it can be used in arithmetic operations, as a symbol it’s good for when you don’t want it to be treated as a number. The unfortunate thing for me/us is that [tabread/write] doesn’t accept symbols, only numbers. Damn! Enter [m_symbolarray] from rjlib.

Using this abstractions you can put symbols into an array in the same way as numbers. With that problem solved I now could decide whether I wanted to jumble words or sentences. In the former case it would be unlikely to return anything that makes sense, which is ok. In the latter case it could still produce interesting results that could fool some people into thinking it was as it should appear. So, it was important to have the option to do both.

Using [textfile] you can specify what a delimiter is. A delimiter specifies where one list item stops. By default [textfile] uses a space as a delimiter. By sending the message [read textfile.txt cr( to [textfile] I can tell it to use a carriage return (Enter key) as a delimiter. Unfortunately there isn’t great documentation on what the others are!

Using the [until] object and a carefully crafted series of operations I crafted a patch to do the following: Decide on delimiter, read the text file, output each list item to the m_symbolarray object, cycle through text file and do this until we reach the end of a text file.

With the table now populated here’s where the interactive part comes in. A user could decide to either dump the contents of the array into a text file or spit out each item to, say, a [text] object for displaying on a screen.

You can download the finished patch below:

Below are my remixes of the two poems for you to download:

it’s a shame that the zine was never finished but it was still a great learning experience.

I’m sure that by now the relatively new [text] object can solve many of the problems I had using [textfile] to read the file, so perhaps at some point in the future there will be an update.