Hey everyone! In place of the usual monthly blog post, today we’ve decided to do something a little different. Very little has happened so far in January and February, and we’d like to talk about that. We’ve been rather quiet on the internals of our game, so we thought it would make sense to talk in-depth about what we’ve been doing behind the scenes. So why don’t you take a seat and share a slice of Pypperoni pizza with us?
Test Server Security
When we released the Test Server, updates were being pushed out at a very frequent pace. Every day for a couple weeks, there would be a new Magic Word here, a new zone there, among many bug fixes. As developers, we had so much fun being able to push out new content at this pace. Never before did we have the opportunity to work on updates and have them playable for testing as soon as we were done with them. It surely motivated us to continue on. Unfortunately, a wrench was thrown in our gears and we soon had a serious problem on our hands.
Because Toontown Offline is a singleplayer game by nature, we’ve never had to deal with security issues before. We aren’t Toontown Rewritten or Corporate Clash with hundreds and thousands of players- we are Toontown Offline. If there’s a district reset, yeah, it’s a big deal and we try to fix it- but it’s not as detrimental to us as it is for other Toontown projects. By opening our Test Server, we were exposing ourselves to the public for the first time. We’ve never had an official, public server before. Never had to. And only a couple weeks after we opened the Test Server, a few malicious individuals took advantage of that fact and began putting our security to the test.
First, our game was dumped. In layman's terms, this means our game was reverse-engineered and it’s source code was published online for anyone to download. In the past, many different Toontown projects have had their games dumped, and we are no exception. However, what made our case special was the nature of Toontown Offline itself. Typically, all Toontown projects have only the client’s code in their executables. Your client is what you see on your screen- a Toon jumping, the gag select screen, those kinds of things. The server on the other hand is typically in charge of keeping everything in check and doing all the important calculations. For us, however, we need to include both the client AND the server code. After all, you wouldn’t be able to host a Mini-Server or run the game in Offline mode without it. So when our game was dumped, that means those who dumped it had access to not only our client code, but server code as well. This made it much, much easier for exploits to be found and abused.
Second, the Test Server was abused. The only malicious behavior our team has ever had to deal with is the moderation of our Discord server. What we were presented with this time was something new and foriegn to us. With access to our client and server code, many exploits were performed on our Test Server. This was both good and bad. Good, because it meant we could patch them up before officially releasing v1.0. Bad, because we didn’t want it to ruin the experiences of our players. At first, we tried some temporary solutions. Our goal was to make sure nobody was coming into the game, spouting foul language and clogging up the server with bots. After that, we would look at assessing the situation and figuring out a more permanent fix. Sadly, things didn’t go so well. The temporary solutions failed to work well, and for a good few weeks we were left frantically scratching our heads. Imagine trying to plug up the holes in a broken pool, but every time you succeed, a new hole pops up. That was us. As a last-ditch effort, we eventually decided to add a new feature- whitelisting. By adding a whitelist to the game, we could control the flow of people coming into our Test Server, thus preventing exploiters. And ever since we’ve added the whitelist, things have been great. It was unfortunate we had to go this route, as it created an extra barrier of entry for entering the Test Server. Regardless, over 600 people have been whitelisted to date, and we couldn’t be more thankful for your patience regarding it. Lucky for us, whitelisting didn’t just serve our purposes for this one situation, but it will also serve as a neat feature for server hosters too. So now, I bet you’re wondering: What is the permanent solution? Well, I’m glad you asked!
Fixing the problem
The crux of our problem was what we use to compile our game. Most Toontown projects out there use a little something called Nirai. This is a deployment tool used for Panda3D games. Up until recently, this is what we’ve used for deploying our game. The upsides are that Nirai is fast, and was relatively easy to set up. As previously stated, we were able to deploy updates insanely fast when we first released the Test Server. A new feature could be finished, and in under 60 seconds it could be released on the Test Server for public consumption. Unfortunately, a downside to Nirai is how increasingly easy it is to dump nowadays. We could’ve chosen to keep Nirai, but pair it with a code virtualization solution. This would essentially secure the game while being able to keep using Nirai, but it would heavily decrease performance. We didn’t want to have to do that, so we decided to leave Nirai behind entirely.
First things first, we wanted to move Toontown Offline from Python 2.7 to Python 3. Support for Python 2.7 is now over, and whatever deployment would switch to would most likely require us to be on Python 3. So before anything else, we had to move to Python 3. We’ve wanted to do this since June 2019, but have often faced setback after setback.
Our goal was to have Toon Parties fully functional before even starting the switch to Python 3. This took way longer than we expected. As explained in previous blog posts, Toon Parties have much more to them than at face value. As such, we first wanted to see if we could get them finished without having to reinvent the wheel. We reached out to a few of our contacts across the community to see if there was any open-source or free-to-use Party code we could use. After all, Parties have probably been developed by half a dozen Toontown projects by now. We found someone who had previously worked on Toon Parties, and they were willing to let us use their code. Great, we thought, that’s a bunch of time saved right there! Now we can focus on other stuff. So we did. We focused on Episodes and other miscellaneous duties while we waited for our source to get back to us with the promised Party code. Weeks and months went past without much follow-up, and we eventually came to the conclusion we wouldn’t be getting any help after all. So on Halloween, we officially started working on Toon Parties ourselves. It sucked to have to reinvent the wheel again, but it had to be done. A little more than a month later, and Toon Parties were complete at the beginning of December. Finally, now that Parties were complete, we could start the move to Python 3.
Only two weeks after that, our lead developer, Little Cat, managed to convert the game to Python 3 almost entirely by himself. So here we are, Episodes done, Parties done, Python 3 done… now what?
Nuitka
Our first attempt to deploy Toontown Offline with something other than Nirai was with Nuitka, a tool that compiles Python to a C or C++ executable. Unlike Nirai, Nuitka wasn’t really aimed at Panda3D games in particular. To our knowledge, no other Toontown project had even tried it before, so we were stepping into new territory. A few weeks passed, and we were encountering countless errors trying to get our game to run with Nuitka. Some files would take up 100 or more gigabytes of ram when compiling, so we had to do extra work just to fix that and the other issues that would arise when trying to compile. At one point, the executable generated was over 300 megabytes in size. 300! That was unacceptable to us. After that, we were only ever able to get the game to show the initial title screen and nothing past it. Eventually, despite many efforts, we decided to scrap all the progress we made with Nuitka and move on to something else- a big slice of Pypperoni pizza.
Pypperoni
Back in June when we were first experiencing our problems with Nirai, we thought about using a compiler called Pypperoni.
This is a project that was made by the developers of The Legend of Pirates Online, or TLOPO for short. TLOPO is the largest private server for what you could consider Toontown’s sister game, Pirates of the Caribbean Online. It closed down alongside Toontown and Pixie Hollow, runs on the same engine, is also written in Python, and uses the same OTP (Online Theme Park) server structure as Toontown. So truth be told, the developers of Pypperoni are well-versed in Panda3D development as are we.
So why didn’t we use it at first? Well, although it sounded great to us, things just didn’t align. Just like Toon Parties, we were promised help with setting it up with Toontown Offline. And again, that unfortunately fell through. As such, we thought that we would have better luck giving Nuitka a go first before coming back to Pypperoni. Well, as you very well know by now, we did end up coming back to Pypperoni. However, this time, we were lucky enough to get a head-start from one of the lead developers of the project. They helped us kick-start the process and we’re now almost done setting up our new Pypperoni deployment.
So far, we’ve gotten the game to run on Linux and Windows, but not Mac. If need be, we may have to delay a Mac release until after the launch of v1.0. We don’t want to delay the release by another month or two just for one platform, while the other two could be playing during that time. We also may end up releasing v1.0 without a feature that was available on our Test Server- Discord integration. This feature is going to need a bit more work before we make it available on the Pypperoni builds of our game. So unless we manage to fix it between now and the release of v1.0, it will most likely be disabled until we can control the issue. Similar to our stance on Mac, we feel that we’ve waited long enough at this point, and would happily release the full version of v1.0 if it means putting one feature on hold.
Where to next?
We still have to do a little bit more testing and finishing up of our Pypperoni builds before we can consider ourselves done with them. Once we’re ready, we plan to release a patch to the Test Server with the brand new build of Toontown Offline, no whitelist enabled. This will be the final test of the game before we close the Test Server and publish the official v1.0 update. Until then though, please hang in there as we work to get that out. We hope you enjoyed reading this special blog post today, especially as it was quite a long read. Until next time, have a good one all!
Benjamin
The Toontown Offline Team
- Toontown Rewritten is a free-to-play revival of Disney's Toontown Online. Create your own Toon and join the battle to save Toontown from the evil robot Cogs. Play now for FREE!
- Sep 22, 2013 The Ice Slide trolley game, from Disney's Toontown Online. Toontown 2003-2013.
- An Archive of Our Own, a project of the Organization for Transformative Works. Fandoms: Toontown Online, Toontown Rewritten - Fandom General Audiences.
- To the Internet Archive Community, Time is running out: please help the Internet Archive today. The average donation is $45. If everyone chips in $5, we can keep our website independent, strong and ad-free. Right now, a generous supporter will match your donation 2-to-1, so your $5 gift turns into $15 for us.
We're at least 4 days from the post Sheriff Cranky made, making accusations towards the TTR team that don't make them look good and certainly don't bode well for future content for the community and TTR has still yet to give an official response as they said they would, at least as far as I have been able to see.
Why is it taking so long for v1.0 to release?Toontown Login
Last month, we explained that our game was now feature-complete. This means that all the features we had planned to be in this version are done, and we are just waiting on bug fixes and more to release the game properly, beyond the Test Server. So, why hasn’t that happened yet? Let’s take a look back to when the Toontown Offline Test Server first opened.![Toontown 2 download Toontown 2 download](https://forum.userstyles.org/uploads/2010/10/toontown%20ryan%20clan%20picture.jpg)
When we released the Test Server, updates were being pushed out at a very frequent pace. Every day for a couple weeks, there would be a new Magic Word here, a new zone there, among many bug fixes. As developers, we had so much fun being able to push out new content at this pace. Never before did we have the opportunity to work on updates and have them playable for testing as soon as we were done with them. It surely motivated us to continue on. Unfortunately, a wrench was thrown in our gears and we soon had a serious problem on our hands.
Because Toontown Offline is a singleplayer game by nature, we’ve never had to deal with security issues before. We aren’t Toontown Rewritten or Corporate Clash with hundreds and thousands of players- we are Toontown Offline. If there’s a district reset, yeah, it’s a big deal and we try to fix it- but it’s not as detrimental to us as it is for other Toontown projects. By opening our Test Server, we were exposing ourselves to the public for the first time. We’ve never had an official, public server before. Never had to. And only a couple weeks after we opened the Test Server, a few malicious individuals took advantage of that fact and began putting our security to the test.
First, our game was dumped. In layman's terms, this means our game was reverse-engineered and it’s source code was published online for anyone to download. In the past, many different Toontown projects have had their games dumped, and we are no exception. However, what made our case special was the nature of Toontown Offline itself. Typically, all Toontown projects have only the client’s code in their executables. Your client is what you see on your screen- a Toon jumping, the gag select screen, those kinds of things. The server on the other hand is typically in charge of keeping everything in check and doing all the important calculations. For us, however, we need to include both the client AND the server code. After all, you wouldn’t be able to host a Mini-Server or run the game in Offline mode without it. So when our game was dumped, that means those who dumped it had access to not only our client code, but server code as well. This made it much, much easier for exploits to be found and abused.
Second, the Test Server was abused. The only malicious behavior our team has ever had to deal with is the moderation of our Discord server. What we were presented with this time was something new and foriegn to us. With access to our client and server code, many exploits were performed on our Test Server. This was both good and bad. Good, because it meant we could patch them up before officially releasing v1.0. Bad, because we didn’t want it to ruin the experiences of our players. At first, we tried some temporary solutions. Our goal was to make sure nobody was coming into the game, spouting foul language and clogging up the server with bots. After that, we would look at assessing the situation and figuring out a more permanent fix. Sadly, things didn’t go so well. The temporary solutions failed to work well, and for a good few weeks we were left frantically scratching our heads. Imagine trying to plug up the holes in a broken pool, but every time you succeed, a new hole pops up. That was us. As a last-ditch effort, we eventually decided to add a new feature- whitelisting. By adding a whitelist to the game, we could control the flow of people coming into our Test Server, thus preventing exploiters. And ever since we’ve added the whitelist, things have been great. It was unfortunate we had to go this route, as it created an extra barrier of entry for entering the Test Server. Regardless, over 600 people have been whitelisted to date, and we couldn’t be more thankful for your patience regarding it. Lucky for us, whitelisting didn’t just serve our purposes for this one situation, but it will also serve as a neat feature for server hosters too. So now, I bet you’re wondering: What is the permanent solution? Well, I’m glad you asked!
Fixing the problem
The crux of our problem was what we use to compile our game. Most Toontown projects out there use a little something called Nirai. This is a deployment tool used for Panda3D games. Up until recently, this is what we’ve used for deploying our game. The upsides are that Nirai is fast, and was relatively easy to set up. As previously stated, we were able to deploy updates insanely fast when we first released the Test Server. A new feature could be finished, and in under 60 seconds it could be released on the Test Server for public consumption. Unfortunately, a downside to Nirai is how increasingly easy it is to dump nowadays. We could’ve chosen to keep Nirai, but pair it with a code virtualization solution. This would essentially secure the game while being able to keep using Nirai, but it would heavily decrease performance. We didn’t want to have to do that, so we decided to leave Nirai behind entirely.
First things first, we wanted to move Toontown Offline from Python 2.7 to Python 3. Support for Python 2.7 is now over, and whatever deployment would switch to would most likely require us to be on Python 3. So before anything else, we had to move to Python 3. We’ve wanted to do this since June 2019, but have often faced setback after setback.
Our goal was to have Toon Parties fully functional before even starting the switch to Python 3. This took way longer than we expected. As explained in previous blog posts, Toon Parties have much more to them than at face value. As such, we first wanted to see if we could get them finished without having to reinvent the wheel. We reached out to a few of our contacts across the community to see if there was any open-source or free-to-use Party code we could use. After all, Parties have probably been developed by half a dozen Toontown projects by now. We found someone who had previously worked on Toon Parties, and they were willing to let us use their code. Great, we thought, that’s a bunch of time saved right there! Now we can focus on other stuff. So we did. We focused on Episodes and other miscellaneous duties while we waited for our source to get back to us with the promised Party code. Weeks and months went past without much follow-up, and we eventually came to the conclusion we wouldn’t be getting any help after all. So on Halloween, we officially started working on Toon Parties ourselves. It sucked to have to reinvent the wheel again, but it had to be done. A little more than a month later, and Toon Parties were complete at the beginning of December. Finally, now that Parties were complete, we could start the move to Python 3.
Only two weeks after that, our lead developer, Little Cat, managed to convert the game to Python 3 almost entirely by himself. So here we are, Episodes done, Parties done, Python 3 done… now what?
Nuitka
Our first attempt to deploy Toontown Offline with something other than Nirai was with Nuitka, a tool that compiles Python to a C or C++ executable. Unlike Nirai, Nuitka wasn’t really aimed at Panda3D games in particular. To our knowledge, no other Toontown project had even tried it before, so we were stepping into new territory. A few weeks passed, and we were encountering countless errors trying to get our game to run with Nuitka. Some files would take up 100 or more gigabytes of ram when compiling, so we had to do extra work just to fix that and the other issues that would arise when trying to compile. At one point, the executable generated was over 300 megabytes in size. 300! That was unacceptable to us. After that, we were only ever able to get the game to show the initial title screen and nothing past it. Eventually, despite many efforts, we decided to scrap all the progress we made with Nuitka and move on to something else- a big slice of Pypperoni pizza.
Pypperoni
Back in June when we were first experiencing our problems with Nirai, we thought about using a compiler called Pypperoni.
This is a project that was made by the developers of The Legend of Pirates Online, or TLOPO for short. TLOPO is the largest private server for what you could consider Toontown’s sister game, Pirates of the Caribbean Online. It closed down alongside Toontown and Pixie Hollow, runs on the same engine, is also written in Python, and uses the same OTP (Online Theme Park) server structure as Toontown. So truth be told, the developers of Pypperoni are well-versed in Panda3D development as are we.
So why didn’t we use it at first? Well, although it sounded great to us, things just didn’t align. Just like Toon Parties, we were promised help with setting it up with Toontown Offline. And again, that unfortunately fell through. As such, we thought that we would have better luck giving Nuitka a go first before coming back to Pypperoni. Well, as you very well know by now, we did end up coming back to Pypperoni. However, this time, we were lucky enough to get a head-start from one of the lead developers of the project. They helped us kick-start the process and we’re now almost done setting up our new Pypperoni deployment.
So far, we’ve gotten the game to run on Linux and Windows, but not Mac. If need be, we may have to delay a Mac release until after the launch of v1.0. We don’t want to delay the release by another month or two just for one platform, while the other two could be playing during that time. We also may end up releasing v1.0 without a feature that was available on our Test Server- Discord integration. This feature is going to need a bit more work before we make it available on the Pypperoni builds of our game. So unless we manage to fix it between now and the release of v1.0, it will most likely be disabled until we can control the issue. Similar to our stance on Mac, we feel that we’ve waited long enough at this point, and would happily release the full version of v1.0 if it means putting one feature on hold.
Where to next?
We still have to do a little bit more testing and finishing up of our Pypperoni builds before we can consider ourselves done with them. Once we’re ready, we plan to release a patch to the Test Server with the brand new build of Toontown Offline, no whitelist enabled. This will be the final test of the game before we close the Test Server and publish the official v1.0 update. Until then though, please hang in there as we work to get that out. We hope you enjoyed reading this special blog post today, especially as it was quite a long read. Until next time, have a good one all!
Toontown Online Archive Download
Benjamin
The Toontown Offline Team
Toontown Online Archives
Toontown
This tag belongs to the Additional Tags Category.
Parent tags (more general):
This tag has not been marked common and can't be filtered on (yet).
Works which have used it as a tag:
Old Toontown Download
The Truth About Toontown by Songvakeppni (SamoShampioni)
Fandoms:Toontown Online
01 Nov 2013Tags
Summary
This is just a silly story that I wrote for a TTRW alpha key competition on Youtube by someone called 'cookingbandtoon'. However, I thought I might as well put it on my AO3 as well. Enjoy! :)The toons and the cogs.. complete opposites in every way. The ten year war has no end in sight. Can anything stop this war? Can the one who caused it, set things right..?- Language:
- English
- Words:
- 1,152
- Chapters:
- 1/1
- Comments:
- 2
- Kudos:
- 14
- Hits:
- 607
Boss Appreciation Day by organicluretrees
Fandoms:Toontown Online, Toontown Rewritten - Fandom
18 Aug 2016Tags
Summary
VP decides to create Boss Appreciation Day to help boost morale within each of the headquarters by allowing the workers to show how thankful they are for their higher ups.Lets see how that turns out on its first run.- Language:
- English
- Words:
- 6,228
- Chapters:
- 5/5
- Kudos:
- 25
- Hits:
- 292
Sneaking into Bossbot HQ by organicluretrees
Fandoms:Toontown Online, Toontown Rewritten - Fandom
05 Sep 2016Tags
Summary
Ozzy the Ambulance Chaser invites Cans the Short Change to come sulk with him at Bossbot HQ in one of the golf courses.- Language:
- English
- Words:
- 2,031
- Chapters:
- 1/1
- Kudos:
- 5
- Hits:
- 167
Called in for a Meeting by organicluretrees
Fandoms:Toontown Online
25 Sep 2016Tags
Summary
Samuel the Backstabber is called in for a special meeting with the CJ to discuss his overall job performance for the past eight years.At least that what he thinks it's for.- Language:
- English
- Words:
- 2,539
- Chapters:
- 1/1
- Kudos:
- 11
- Hits:
- 191
The Evil Life by entercleverpennamehere
Fandoms:Disney - All Media Types
13 Apr 2018Tags
Summary
What do you get if you cross Disney Villains with communal living? A sitcom goldmine! See sparks fly and tempers flare as the Villains cope with everyday life in a shared loving space in the new smash-hit sitcom ‘The Evil Life’!- Language:
- English
- Words:
- 4,230
- Chapters:
- 2/?
- Comments:
- 3
- Kudos:
- 43
- Bookmarks:
- 2
- Hits:
- 620
The Scrapyard by organicluretrees
Fandoms:Toontown Online
31 Jul 2018Tags
Summary
A quick look into a place where most cogs would want to forget it exists.- Language:
- English
- Words:
- 1,020
- Chapters:
- 1/1
- Kudos:
- 5
- Hits:
- 134
Nuttyberry's Adventures In Toontown by orphan_account
Fandoms:Toontown Online
03 Nov 2015Tags
Summary
A human has been sucked into the world of Toontown through his computer. Now he plays the role of his toon character in the game, Nuttyberry. Adventures await him and his new friends he makes in Toontown!- Language:
- English
- Words:
- 564
- Chapters:
- 1/?
- Comments:
- 1
- Kudos:
- 2
- Bookmarks:
- 1
- Hits:
- 34
Incandescence by Made_Of_Love
Fandoms:Kingdom Hearts, Disney - All Media Types
06 Mar 2020Tags
Summary
After the events of Kingdom Hearts the worlds have returned.Except yours..For nineteen years you lived on an artificial world. But darkness has forced you from your home. Now you must go on an adventure to get your home back and learn the dark secrets behind its disappearance while trying to avoid appearing on the Organization's radar. And that's easier said than done when you keep finding yourself getting involved with a certain red-haired assassin.- Language:
- English
- Words:
- 443,168
- Chapters:
- 65/?
- Comments:
- 136
- Kudos:
- 464
- Bookmarks:
- 26
- Hits:
- 4713