Developer Update File system changes and backwards compatibility

[Seiichi]

Since the last development update, Dragon has been working hard on redesigning the file system. We originally had planned against this but after running into an issue with one of our final tasks found it was unavoidable, and that we would need to start from the ground up.

Here are the points we had to consider while weighing our options:

• Packed vs unpacked regarding performance, security, and patch generation
• Size optimization vs speed optimization
• Backward compatibility

The biggest bottleneck in Dragon's Prophet is how it accesses and manages the packed data format it currently uses in the client. In most cases, the use of packed data with index files leads to better performance by allowing faster access to CPUs/GPUs. This is not the case with how Dragon's Prophet is developed as it leads to a significant bottleneck, especially when the user interface is involved. Examples of this are the map or any interface that has a significant amount of data which leads to lower FPS, poor loading times or stalling. Going with an unpacked system would alleviate this bottleneck but introduces issues with security as well as slower accessing of files (primarily textures) especially with older or slower hardware. Dragon was able to find a solution that alleviates the bottleneck while also supporting good overall performance. It also allowed us to keep the usage of crunch, a texture compression library, that we have found is needed for players who experience high latency or use older hardware.

With the new system decided we now had to choose whether we would focus on performance, or a higher level of compression. Higher compression would decrease the footprint of the client but would also bring higher CPU load which could affect performance given how Dragon's Prophet is designed. Our first test resulted in a 20% increase in packed data size (about 3.5GB) with almost no impact on performance. After playing with our implementation and options we were able to lower that down to a 7.6% increase in packed data size (about 1.4GB) with no impact on performance. All this testing was primarily done on HDD as that will be one of the primary bottleneck's players will experience; switching to an SSD could lead to a significant increase in performance.

The final decision we had to make was on supporting backwards compatibility, and how far back we are logically able to support. Dragon's Prophet was designed to work best with Windows 7 and hardware between 2008-2012, which is why there are performance issues with anything more modern. After consideration we decided to drop support for any CPU that is older than Intel: Sandy Bridge or AMD: Bulldozer, both of which were released in late 2011 but supply access to the AVX instruction set. From all that I have seen, most if not all the players should be unaffected.

Because there are going to be significant changes to the file system and how data is going to be packed and managed, we are also having to rework our patching system (again) as well as develop a new launcher. The first version of the launcher will be basic, but it will be based off Microsoft's WebView2 platform allowing us to build a web page and use that for the interface, which makes it extremely easy to make changes or adjustments as needed. It will also let us incorporate a first-run system to allow us to decouple data from the installer so that we can move to an actual installer and then download data from the launcher properly.

We are still completing changes to our patching process, the launcher, and other miscellaneous tasks. When these are completed, we will be releasing the update to our testers and hopefully moving on to a public beta. A new client will be needed for the update due to the major differences between them, as well as a client specifically for the test server.

The news background was made by @Vipera.

 

[Billy]

Ahhh I'm so excited to see how this goes! Better optimization/preformance within the next 4 years would be a neat thing, if it's possible within that time. Thankfully too, most computers with hardware that old doesn't work anymore, so yeah there def shouldn't be many ppl affected by that

 

[Seiichi]

The file system changes have been completed, barring any adjustments required by the launcher. We're mostly just working out any quirks the launcher will have, and will then look to finishing that up.

 

[Seiichi]

We think we have worked out most of the issues with the file system changes and the new launcher that we have made, and we will be moving to doing some internal testing before letting beta testers try out the new client. If no significant issues are found in these two phases, we will move to an open beta to make sure there are no major incompatibilities or bugs that require fixing.

 

[Seiichi]

So there have obviously been some delays, as we ran into some changes needing to be made to how files were handled, and also how the patch system was being done.

Originally, we were going to use a repair file to track historical changes... Until we realized that generating the current patch in the new system creates a 100mb file, with about 2.6million lines... Which would have to be downloaded everytime there was a version check. This required re-thinking how the patch system was handling historical version changes, which took a bit longer because DP has an excessive number of files (~560,000).

We also went back and fixed an issue which could lead to possible file corruption, and issues with locking up due to I/O. We also lowered RAM and CPU usage as much as possible, which was an added benefit.

Now we are having to move on to getting the changes into the launcher and getting the launcher (and backend for the launcher) prepared for testing. Because web development is not what Dragon enjoys, and he has been dealing with some health issues I cannot tell you how long it will be before we release it to beta testers. Things are being worked on, however, just a time-consuming process to make sure we do things well and with as few complications in the future.

With the new launcher, players will only need to install the launcher which will then download and install the client. It will also handle downloads (obviously) but will allow for automatic repairs for any modified files (outside of configuration files) or corrupted files. In theory, it should completely negate half the support issues that tend to happen with Dragon's Prophet, we hope.

 

[Sylvester]

Hi, thank for the news.