RescindentArchives Status Update and new Roadmap

Status Update

 It has been some time since I wrote blog entry because of life becoming busy so I'll try to summarise the current status of the RescindentArchives project. The channel was brought back online in May after various events made it difficult to host and it has been largely stable since. There have been a few issues with playback freezing and a bug or two with the show scheduling, which led to the decision to rewrite the bot with more familiar technologies. The changes to audio and music playback are still on the roadmap once the rewrite is complete.

RescindentArchives will remain a passion project I work on in spare time and I have shelved plans to monetise the channel in any way.

Roadmap

0.5.0 - "Rewrite"

• Complete rewrite of the channel's bot
• Improved archived management to reduce complexity of adding videos and categories
• Addition of new content from 2020 and 2021
  

0.5.1 - "Show variety"

• Improved show scheduling
• Time-based show slots to vary content during the day

 0.5.2 - "Soundtrack"

• Music selection based on what is playing
  

 

RescindentArchives is back on a brand new server!

This will be a short post, but a necessary one. The RescindentArchives stream is back online and development shall begin anew, supported by a new server.

Why did RescindentArchives disappear in the first place? There were a number of personal and technical reasons, including a few impending drive failures. After much thought I consolidated all hardware resources into a brand new server that will serve for 5-10 years, as my old server has done.

What next? Development will continue, albeit differently from before since time I have different priorities now. 0.4.2 will be released, but the rate of addition to the audio catalogue will be an ongoing issue. The next major version, 0.4.3, will focus on adding overlay-related features and enhancements.

There are some other plans I have in mind but they are very much in the planning phase! Otherwise feel free to take a look at the newly-restored RescindentArchives stream at the following sites:

• https://www.twitch.tv/rescindentarchives
• https://dlive.tv/RescindentArchives 
  

RescindentArchives 0.4.2 (Phase One) Release Notes

0.4.2 (Phase One) - "Does that track really fit the video?"

 

0.4.2 is a big release in a number of ways so it has been broken down into two 'phases'. Phase One upgrades the existing software to support new features and introduces some fixes, with Phase Two following in the next few weeks. Releasing these changes separately also has benefits such as ensuring they are resilient to outages and avoiding a big release of multiple components (what I used to term a 'Big Bang' release).

• Added integration with systems that will be released in Phase Two
• Updated the schedule to include some more categories and events, as well as eliminating the possibility of alternating shows (i.e. Warcraft, Warships, Warcraft, Warships etc.)
• Resolved potential issues caused by unnecessary caching

0.4.1 Release Notes

0.4.1 - "To mute or not to mute"

The heart of 0.4.1 is making the video audio available where possible, involving the deployment of a database and development of management tools. Smaller changes to improve the variety of content on the channel were made and some forgotten videos return in this version.

• Video audio will play for videos where possible. A number of videos have been enabled to play audio, many more will be reviewed and enabled in the coming days
• Play order algorithms now exist to add some randomness or to improve the variety of games played for events (genres etc)
• Daily Twitter announcements are made to indicate what is playing
• Some videos that were outside the schedule, termed ‘Wild Cards’, will now play as part of a special weekly event

Other notable changes

Behind the scenes there have been many changes to the way that the stream's bot code is built and deployed. The build and deployment process for beta versions is now completely automated, meaning more time spent on development and less on building and packaging everything.

April Progress Update

It has been a long time since I updated this blog and there are a lot of changes to discuss, so I shall provide a brief summary and then you can read whatever interests you. Otherwise, I hope you are staying safe and well in these difficult times!

23rd March: RescindentArchives returned with the 0.4.0h release
30th March: Archive server drive replacement 1
6th April:

• Archive server drive replacement 2
• Broadcaster server CPU and RAM upgrade

13th April: Archive server drive replacement 3

Wait- What about 0.4.1?

0.4.1 is releasing on 20th April, with support for playing audio on select videos and numerous enhancements. Many extra changes were required to bring this version to fruition, more discussion on that next week.

RescindentArchives returns and 0.4.0h

It was with some great relief that my family and I found somewhere to settle down for a number of months, not a moment too soon given the situation worldwide. This gave me reason to bring the stream back online, but before that could happen I had to solve several problems:

1. Setup the stream on a much less powerful server
2. The new server I had with me would not run 0.4.0g
3. Streamlabs OBS was very unstable on the new server

The first problem was solved with time and the second did not go away so I had to abandon the 0.4.0g build I had and fix the issues with a fresh build, 0.4.0h. This would not have been possible without the code residing in a Version Control System (VCS), Git in my case. In the months since I last streamed, back in late January, it would seem Streamlabs OBS became more unstable and kept dropping every few hours. Instability of this sort is something I strive to avoid, so I looked to OBS Studio for an answer- with the WebSocket package it ran without incident and was easier to integrate with than Streamlabs.

So all fine, the stream was back, correct? Not quite. The new server was an old late 2015 Macbook Pro running Windows 10 and could handle the encoding for hours before thermal throttling kicked in and meant even 720p at 30 FPS would cripple it. I was fortunate to have brought my group of servers along and stored them, so I dug out the old server, wired it up, installed OBS Studio and ran 0.4.0h. RescindentArchives was back!

Archive Server drive replacement

The Archive Server is the machine holding my archived video, holding it on a ZFS volume with multiple mirrors of 2 disks. This was setup in April 2017 with some new Western Digital Black Hard Disk Drives (HDDs), so it has been three years since they began spinning 24/7. Three years is a long time in a HDDs lifetime, as the excellent reports by BackBlaze attest to, so I decided to swap out a drive in each mirror to reduce the chances of data loss in the near future. Replacing each drive takes around 8 hours, which is why I had to take the stream down each Monday to facilitate the migration.

There are other drive setups, RAIDZ2 and RAIDZ3, that can reduce the capacity penalties of mirroring, but they can only handle 2 or 3 drive failures respectively. There are some excellent blogs out there endlessly debating the merits of different array setups and I opted for mirroring since the worst case is I lose some data rather than all of it.

Broadcast Server CPU and RAM upgrade

Upgrading the Broadcast Server was an unscheduled change, spurred on by the discovery of an old AM3+ motherboard with 16GB RAM and an FX-8350. All I needed was a new CPU cooler, which arrived a day early (thanks Amazon!).

What does 0.4.0g2 actually mean?

What does 0.4.0g2 actually mean?

Viewers will have seen these versions on the stream's title and likely wondered if they had any meaning. Fear not, they do and the version format has evolved to meet various demands! While I began with a vague notion of what the version numbers meant, this changed during 0.4 to include the notion of 'features' and 'fixes'. This is the current meaning behind those numbers and letters:

<project version>.<code branch>.<feature branch><iteration><fix version>

project version: A number. Increment this whenever the project has reached a big milestone (more discussion on this in the future
code branch: A number. Which branch of the codebase this version represents. Versions within the same branch will have few structural changes. 0.4 introduced sweeping changes to the structure of the codebase, 0.5 will do the same
feature branch: A number. Each 'feature branch' has the objective of adding a new feature or changing an existing one. 0.4.0 changed video scheduling, 0.4.1 will change when video muting/ unmuting takes place
iteration: A letter. An iteration on the feature being worked-on
fix version: A number. Reserved for releases that only introduce fixes

So, in summary 0.4.0g2 means...

0: The stream video player bot is not what I would consider a full 'release' at this time, for various reasons
4: The codebase received a huge rework since the last release of 0.3 and was incremented to 0.4
0: The first feature release of the 0.4 code branch
g: The seventh iteration on the 0.4 branch
2: The second fix-only release of 0.4.0g

Why include the version number in the title?

The version number is a reminder to myself and an indication to eagle-eyed viewers that something changed. Stream titles do not give much room for description, so the version numbers lend themselves better than, "Fixed the scheduler so it does not reset playback"!

There is a time rapidly approaching when a single version number will not hold-up, but that is a tale for another time!

Reflecting on 0.4 and the future

After the busy holiday season it’s a good time to take stock of what happened with the project recently. Further down I discuss the lessons my future self should remember and discuss the roadmap. This is a summary of developments during the 0.4 branch:

30.10: 0.4.0 features and 0.4.0a fixes
31.10: 0.4.0b features and fixes
07.11: Server migration
11.11: Server maintenance (the infamous PSU issue)
12.11: 0.4.0c features and fixes
18.11: 0.4.0d features (downtime scheduler)
20.11: Content update
02.12: 0.4.0e features (daily shows scheduled)
09.12: 0.4.0f features (series slots added)
13.12: 0.4.0f fixes (effectively 0.4.0f1)
16.12: Outage begins after server relocation
21.12: Restoration after reversing server relocation
23.12: 0.4.0g features (support for fixed-date shows and weekly “themes”)
24.12: 'Countdown to 2020' event begins
01.01: 'Countdown to 2020' event ends
12.01: 0.4.0g1 fixes
15.01: 0.4.0g2 fixes

Lessons for 2020

• Listen to your own feelings about the project, especially when it’s criticism
• Don’t be afraid to abandon features that don’t work
• When things go wrong, consider if it’s catastrophic or manageable
• Make sure your version format can support every eventuality
• Work out a balance between feature and content releases
• If it isn’t broken, don’t fix/ change/ touch it without good reason!

Roadmap update

0.4.1 - To mute or to unmute?
0.4.2 - Vote for your soundtrack Does that track really fit the video?
0.4.3 - To be announced
0.4.4 - To be announced
0.4.5 - To be announced
0.5.0 - To be announced (hint: involves a lot of rebuilding)

In developing the scheduling system I learnt a great deal about what works, for the stream and my development process. After developing the original format, new shows every 30 minutes, I found that replaying the same content at short intervals became boring for me and the voting system was never used. It is never nice to admit something you created is bad, but accepting this feeling and being unafraid to embrace change can be good (as long as we also accept perfection is unrealistic and that there is a logical end to this struggle). If I am bored with what’s playing, it’s certain that others will be as well. So, 0.4.0b was my initial response and from there I kept iterating on the format. Updating the on-screen schedule’s format, removing the voting system and toggling what’s shown on the information panel were the simple changes. Storing state and continuing shows at their previous point, episodic content played at particular times, daily shows and fixed-date shows were challenging to implement, but ultimately improved upon the purpose of the stream.

The version numbers the stream had gone through may look strange, but the format reflects how I develop software. The agile methodology has been my favoured approach for years and it shines when you need to iterate- the final version in the 0.4 branch is radically different from the first. When something went wrong the first consideration was time- do I fix it now, wait until the next version or is is manageable in the short-term? Some of these decisions worked out well, others not so much. Fixing some playback issues in the last few days was absolutely the right call, but sticking with Streamlabs OBS lead to many stream crashes. However, Streamlabs OBS has broadcast my stream for many hours at a time and its suite of tools gave the stream a headstart where overlays are concerned. There is a technical problem that consumed time, but was there a good reason to invest time testing other streaming software? My answer came once I was satisfied with 0.4- the next version will take more time than the previous iterations, use it to test other streaming software too! So, no there was no good reason before, but now is the perfect time.

The roadmap is still in place and I am working on 0.4.1 and 0.4.2 simultaneously because they both demand a better-structured storage system. 0.4.2 will now aim to harmonise the video and music, instead of having a completely random playlist system. I have an idea of priorities for 0.4.3-0.4.5, but I will announce them once I have 0.4.1 released. All of this work is being done with objectives extending to 0.5, but rest assured this work will have little effect on the existing stream's bot. Vetting, compressing and adding material to the archive has been a time-consuming process that would benefit from some automation.