RescindentArchives 0.4.2 (Phase One) Release Notes

0.4.2 (Phase One) - "Does that track really fit the video?"

 

0.4.2 is a big release in a number of ways so it has been broken down into two 'phases'. Phase One upgrades the existing software to support new features and introduces some fixes, with Phase Two following in the next few weeks. Releasing these changes separately also has benefits such as ensuring they are resilient to outages and avoiding a big release of multiple components (what I used to term a 'Big Bang' release).

• Added integration with systems that will be released in Phase Two
• Updated the schedule to include some more categories and events, as well as eliminating the possibility of alternating shows (i.e. Warcraft, Warships, Warcraft, Warships etc.)
• Resolved potential issues caused by unnecessary caching

0.4.1 Release Notes

0.4.1 - "To mute or not to mute"

The heart of 0.4.1 is making the video audio available where possible, involving the deployment of a database and development of management tools. Smaller changes to improve the variety of content on the channel were made and some forgotten videos return in this version.

• Video audio will play for videos where possible. A number of videos have been enabled to play audio, many more will be reviewed and enabled in the coming days
• Play order algorithms now exist to add some randomness or to improve the variety of games played for events (genres etc)
• Daily Twitter announcements are made to indicate what is playing
• Some videos that were outside the schedule, termed ‘Wild Cards’, will now play as part of a special weekly event

Other notable changes

Behind the scenes there have been many changes to the way that the stream's bot code is built and deployed. The build and deployment process for beta versions is now completely automated, meaning more time spent on development and less on building and packaging everything.

April Progress Update

It has been a long time since I updated this blog and there are a lot of changes to discuss, so I shall provide a brief summary and then you can read whatever interests you. Otherwise, I hope you are staying safe and well in these difficult times!

23rd March: RescindentArchives returned with the 0.4.0h release
30th March: Archive server drive replacement 1
6th April:

• Archive server drive replacement 2
• Broadcaster server CPU and RAM upgrade

13th April: Archive server drive replacement 3

Wait- What about 0.4.1?

0.4.1 is releasing on 20th April, with support for playing audio on select videos and numerous enhancements. Many extra changes were required to bring this version to fruition, more discussion on that next week.

RescindentArchives returns and 0.4.0h

It was with some great relief that my family and I found somewhere to settle down for a number of months, not a moment too soon given the situation worldwide. This gave me reason to bring the stream back online, but before that could happen I had to solve several problems:

1. Setup the stream on a much less powerful server
2. The new server I had with me would not run 0.4.0g
3. Streamlabs OBS was very unstable on the new server

The first problem was solved with time and the second did not go away so I had to abandon the 0.4.0g build I had and fix the issues with a fresh build, 0.4.0h. This would not have been possible without the code residing in a Version Control System (VCS), Git in my case. In the months since I last streamed, back in late January, it would seem Streamlabs OBS became more unstable and kept dropping every few hours. Instability of this sort is something I strive to avoid, so I looked to OBS Studio for an answer- with the WebSocket package it ran without incident and was easier to integrate with than Streamlabs.

So all fine, the stream was back, correct? Not quite. The new server was an old late 2015 Macbook Pro running Windows 10 and could handle the encoding for hours before thermal throttling kicked in and meant even 720p at 30 FPS would cripple it. I was fortunate to have brought my group of servers along and stored them, so I dug out the old server, wired it up, installed OBS Studio and ran 0.4.0h. RescindentArchives was back!

Archive Server drive replacement

The Archive Server is the machine holding my archived video, holding it on a ZFS volume with multiple mirrors of 2 disks. This was setup in April 2017 with some new Western Digital Black Hard Disk Drives (HDDs), so it has been three years since they began spinning 24/7. Three years is a long time in a HDDs lifetime, as the excellent reports by BackBlaze attest to, so I decided to swap out a drive in each mirror to reduce the chances of data loss in the near future. Replacing each drive takes around 8 hours, which is why I had to take the stream down each Monday to facilitate the migration.

There are other drive setups, RAIDZ2 and RAIDZ3, that can reduce the capacity penalties of mirroring, but they can only handle 2 or 3 drive failures respectively. There are some excellent blogs out there endlessly debating the merits of different array setups and I opted for mirroring since the worst case is I lose some data rather than all of it.

Broadcast Server CPU and RAM upgrade

Upgrading the Broadcast Server was an unscheduled change, spurred on by the discovery of an old AM3+ motherboard with 16GB RAM and an FX-8350. All I needed was a new CPU cooler, which arrived a day early (thanks Amazon!).

What does 0.4.0g2 actually mean?

What does 0.4.0g2 actually mean?

Viewers will have seen these versions on the stream's title and likely wondered if they had any meaning. Fear not, they do and the version format has evolved to meet various demands! While I began with a vague notion of what the version numbers meant, this changed during 0.4 to include the notion of 'features' and 'fixes'. This is the current meaning behind those numbers and letters:

<project version>.<code branch>.<feature branch><iteration><fix version>

project version: A number. Increment this whenever the project has reached a big milestone (more discussion on this in the future
code branch: A number. Which branch of the codebase this version represents. Versions within the same branch will have few structural changes. 0.4 introduced sweeping changes to the structure of the codebase, 0.5 will do the same
feature branch: A number. Each 'feature branch' has the objective of adding a new feature or changing an existing one. 0.4.0 changed video scheduling, 0.4.1 will change when video muting/ unmuting takes place
iteration: A letter. An iteration on the feature being worked-on
fix version: A number. Reserved for releases that only introduce fixes

So, in summary 0.4.0g2 means...

0: The stream video player bot is not what I would consider a full 'release' at this time, for various reasons
4: The codebase received a huge rework since the last release of 0.3 and was incremented to 0.4
0: The first feature release of the 0.4 code branch
g: The seventh iteration on the 0.4 branch
2: The second fix-only release of 0.4.0g

Why include the version number in the title?

The version number is a reminder to myself and an indication to eagle-eyed viewers that something changed. Stream titles do not give much room for description, so the version numbers lend themselves better than, "Fixed the scheduler so it does not reset playback"!

There is a time rapidly approaching when a single version number will not hold-up, but that is a tale for another time!

Reflecting on 0.4 and the future

After the busy holiday season it’s a good time to take stock of what happened with the project recently. Further down I discuss the lessons my future self should remember and discuss the roadmap. This is a summary of developments during the 0.4 branch:

30.10: 0.4.0 features and 0.4.0a fixes
31.10: 0.4.0b features and fixes
07.11: Server migration
11.11: Server maintenance (the infamous PSU issue)
12.11: 0.4.0c features and fixes
18.11: 0.4.0d features (downtime scheduler)
20.11: Content update
02.12: 0.4.0e features (daily shows scheduled)
09.12: 0.4.0f features (series slots added)
13.12: 0.4.0f fixes (effectively 0.4.0f1)
16.12: Outage begins after server relocation
21.12: Restoration after reversing server relocation
23.12: 0.4.0g features (support for fixed-date shows and weekly “themes”)
24.12: 'Countdown to 2020' event begins
01.01: 'Countdown to 2020' event ends
12.01: 0.4.0g1 fixes
15.01: 0.4.0g2 fixes

Lessons for 2020

• Listen to your own feelings about the project, especially when it’s criticism
• Don’t be afraid to abandon features that don’t work
• When things go wrong, consider if it’s catastrophic or manageable
• Make sure your version format can support every eventuality
• Work out a balance between feature and content releases
• If it isn’t broken, don’t fix/ change/ touch it without good reason!

Roadmap update

0.4.1 - To mute or to unmute?
0.4.2 - Vote for your soundtrack Does that track really fit the video?
0.4.3 - To be announced
0.4.4 - To be announced
0.4.5 - To be announced
0.5.0 - To be announced (hint: involves a lot of rebuilding)

In developing the scheduling system I learnt a great deal about what works, for the stream and my development process. After developing the original format, new shows every 30 minutes, I found that replaying the same content at short intervals became boring for me and the voting system was never used. It is never nice to admit something you created is bad, but accepting this feeling and being unafraid to embrace change can be good (as long as we also accept perfection is unrealistic and that there is a logical end to this struggle). If I am bored with what’s playing, it’s certain that others will be as well. So, 0.4.0b was my initial response and from there I kept iterating on the format. Updating the on-screen schedule’s format, removing the voting system and toggling what’s shown on the information panel were the simple changes. Storing state and continuing shows at their previous point, episodic content played at particular times, daily shows and fixed-date shows were challenging to implement, but ultimately improved upon the purpose of the stream.

The version numbers the stream had gone through may look strange, but the format reflects how I develop software. The agile methodology has been my favoured approach for years and it shines when you need to iterate- the final version in the 0.4 branch is radically different from the first. When something went wrong the first consideration was time- do I fix it now, wait until the next version or is is manageable in the short-term? Some of these decisions worked out well, others not so much. Fixing some playback issues in the last few days was absolutely the right call, but sticking with Streamlabs OBS lead to many stream crashes. However, Streamlabs OBS has broadcast my stream for many hours at a time and its suite of tools gave the stream a headstart where overlays are concerned. There is a technical problem that consumed time, but was there a good reason to invest time testing other streaming software? My answer came once I was satisfied with 0.4- the next version will take more time than the previous iterations, use it to test other streaming software too! So, no there was no good reason before, but now is the perfect time.

The roadmap is still in place and I am working on 0.4.1 and 0.4.2 simultaneously because they both demand a better-structured storage system. 0.4.2 will now aim to harmonise the video and music, instead of having a completely random playlist system. I have an idea of priorities for 0.4.3-0.4.5, but I will announce them once I have 0.4.1 released. All of this work is being done with objectives extending to 0.5, but rest assured this work will have little effect on the existing stream's bot. Vetting, compressing and adding material to the archive has been a time-consuming process that would benefit from some automation.

RescindentArchives - VLC, OBS, NVENC and PSUs

If you had wandered into my archive stream at any point in the last few days you would notice how it was starting, freezing, ending and resetting. I apologise for this downtime and subsequent loss of time from a viewer's perspective, but the intention of this post is to provide a hint to anyone that encounters the problem I had experienced.

Why is VLC media player freezing/ crashing and OBS crashing unexpectedly?

There could be any number of reasons, but in my case there were no logs or crash dumps to provide an explanation. After trying several nvidia drivers I trust to be stable I stumbled upon a forum discussing the possibility of their PC’s Power Supply Unit (PSU). Unable to find a software solution to my issue I thought, "what the hell", and dug out a PSU from a server I am repairing. I write this post at 8pm, 7 hours after going live, and neither VLC nor OBS have crashed. No, I did not publish this after writing it... just in case!

Answer: When nothing else works, try changing your Power Supply Unit (PSU)!

RescindentArchives server migration 2019/11/05 to 2019/11/07

Today the RescindentArchives stream was migrated from a Virtual Machine to a physical server. There were a number of reasons that I shall discuss in this post.

Honourable technology mentions before I begin...

• VMWare ESXi, a bare-metal hypervisor I use to test many Virtual Machines
• Nvidia and AMD graphics cards I use for streaming video

Virtual Machines vs Physical Machines

Virtual Machines (VMs) are a fascinating technology that are often paraded as the answer to every system administrator's woes, but there are downsides to using them. VMs, in most configurations, can be backed-up in entirety, moved from one data centre to another and their performance boosted by moving them to more capable hosts. The hosts are the main reason I am hesitant to use VMs for anything performance or time-sensitive- often a customer will have no idea what other VMs are running on the host. These other VMs have their own demands and in some cases I have seen troubled VMs drain enough resources to cripple their kindred.

You may be asking me why I chose to use a Virtual Machine in the first place, a good question! I love experimenting with technology! When I began the journey of setting up RescindentArchives I wanted to try out VMWare ESxi's GPU passthrough and managed, after some troubles, to get my Nvidia Geforce GTX 1050Ti to work inside a Windows 10 VM. The downside of this is the VM is now tied to its host, eliminating most of the benefits of using a VM in the first place.

Development environments

In my time developing software professionally it was drilled into me from day one that Production was not our platform- it was the domain of our users and support team. As Developers it was our role to deliver improvements, observe problems, work out solutions and pass these all onto the support team, who would scrutinise and potentially implement them. I stand by this philosophy, but when you are the only person managing a project like RescindentArchives you become the Developer and support team! So, by designating a new physical server as Production I can now use the VM for extensively testing new versions before they go live.

So what is this mystical new physical server?

I have loved building new machines since I was in my early teens, fortunately one of them was sitting around not doing much. The CPU is still going 10+ years after it was built, seeing me through secondary school and University, since combined with a new motherboard and hard drive. During testing I discovered how far GPU encoding has come- the Radeon R7 250X GPU I tested could not cope with 1080p at 60fps. As a result I swapped the 1050 Ti into the new host and the 250X into the old. This will cause a performance disparity, but I will resolve this when extra parts become available.

The Virtual Machine formerly hosting RescindentArchives runs on my main Virtual Machine host, a server called, "Contrivian", that has served me well for 7 years:

• AMD FX-8350 (8-core, 4.0Ghz)
• 16GB DDR3 RAM 
• Nvidia Geforce GTX 1050 Ti

These are the new RescindentArchives hosts

 

Rescindent (Production)

• Intel Core 2 Quad Q9550 (4-core, 2.8Ghz)
• 4GB RAM
• Nvidia Geforce GTX 1050 Ti

I am already seeing much better networking performance with this physical server.

VW10/ Contrivian (Development)

• 3 vCPUs allocated from AMD FX-8350 (8-core, 4.0Ghz)
• 6GB RAM allocated (host max. 16GB DDR3 RAM)
• AMD Radeon R7 250X

The development server will be back in action soon.