How do I work around hitting the inode limit imposed by web hosts?

Question

I am currently trying to find a host for my website. The website allows people to host projects in a somewhat similar way to this. I would rather not reveal more about what it is until I release it. The problem is, that with users uploading files for their mods/projects/etc I would quickly reach any inode limit, especially with my current folder structure.

http://www.example.com/files/HOrrZCINYUoTxPVzaRrRFYUQRDxPPRun/wiD4Pj38Tkq/SomeFileUploadedWithTheProperNameStillHere.jar.

As you can see from that, it is a quite long URL. The first set of jumble is the project id (every project has one), it stays the same for all files of the same project. The second set of jumble is the file id, randomly generated for each file. There is a file id so that files of the same name can be uploaded to each project (Example would if someone uploads 2 MyProject.jar files, one for version 1 and another for version 2). So far I cannot find any hosts that allow for unlimited inodes, every host has a 25,000-500,000 limit (hence too small). Even if that was big enough, I would prefer not to have the imminent loom of inode limit bypassing over my head all the time (It would drive me insane). I would like either a link to a host with unlimited MySQL, disk space, and inodes. Or another way to do this. I was thinking about storing the files in a MySQL database, however it seems like most hosts limit those too (Including the database's files in the inode count).

Answer 1

The inode limit is only set on things such as reseller/shared hosting. If you have a VPS you can expect to be limited to around 1,000,000 inode and if you have a dedicated server you should be limited. However as you know, this will cost more depending on your needs.

They have to limit the inodes, if they didn't limit this you would be looking at "unlimited resources". Everytime you upload an item/make an insert into the database you will use an inode. Think about it, how big are your files going to be? Basing this just on uploads and not on database inserts if your limit is 25,000 and your average upload is 5mb this would equate to 125000mb which is about 122gb this is on the bottom scale. If you look at the size of the MySQL jar library which is 6.1mb and take a mid-range of the limit you have found, call it 240,000 inodes then thats 1464000mb, which is about 1429gb which is 1.39tb.

Surely if you are looking at something that would use up this amount of resources this quickly you would need to look at multiple servers, multiple locations, dedicated database servers and alot of bandwidth. The limit is all about not eating up server resources but if you do wish to implement this it maybe a case where the best possible solution is for you to host this yourself, but bare in mind for constant uploading of large files you will require a very good upload speed connection to the server.

Answer 2

Outside of "cloud" hosting, no resources are unlimited, regardless of what a marketing department says. If you need a large amount of resources, expect to pay for a large amount of resources. Shared hosting is economy hosting, and intended to be a cheap service for users who are just getting started with hosting, and want to try out a simple site or two. Don't expect to produce a large application on this type of hosting.

There are many reasons why a host will limit the amount of inodes available. One reason is that they are actually not unlimited on a physical server. Many of the popular filesystems, such as ext3 and ext4, have inode limits. With those in particular, you need to determine the total number of inodes, and thus the static space required, when you crate the filesytem. Each inode takes up space. Therefore, the more inodes you use, the less disk space will be availalbe. Fortunately, these are small, so you can allocate many of them, and that's fine as long as they are used reasonably. These are typically a number determined as a fraction of the total disk space available. On shared environments, these must be distributed among the users on the server so that nobody can crash the whole server by creating tons of empty files.

Another reason that a host may limit inodes is because they don't want to encourage certain types of storage. Email is an example where you typically can have very many small files, and large amounts of inode usage can indicate a spammer. Also, consider that some hosts will also not like strange data storage schemes, and could be a sign that something problematic is happening. Or perhaps they want to create some reasonable way of limiting "unlimited" disk space usage.

Usually, unless you have a lot of tiny files, you couldn't use that many inodes without also using a lot of space, and also a lot of other resources as well. If your files are small, then consider some other data storage method, such as a database. If the data is actually larger, then expect to pay for more disk usage.

Finally, if your main concern isn't the cost, but wanting to be sure that your application is scalable, then you could look into some of the more scalable solutions. For instance, there's the Amazon elastic hosting and also the S3 storage buckets. They are designed to allow for increasing resources as needed. However, you will need still need to pay for all of your usage, so if you do end up using a lot of resources, it will still cost money.

Answer 3

I can't find any reliable host with no limit on inodes on there VPS plans. I was having this inodes problem on Hostgator and in-fact they have very low inodes limit if you are a reseller and hosting multiple sites. So after researching a lot I decided to go with Known Host which also has inodes limit but that is really very high and with different VPS packages it also changes. After using hostgator I was very reluctant to move to any other host but I found Known Host really very good.

Answer 4

I have had to shut down 2 websites that were doing very well as far as members and traffic. I was starting to look into cashing in on advertisement until I got an email warning me that I reached my max for storage and inodes. I lost all my members and couldn't start over.

I realized that the only way to have a website with unlimited inodes, storage and resources similar to facebook, youtube and other social media is to have my own servers set up at a warehouse with engineers on payroll to support software upgrades and server issues.