Upload large files + progress bar

[eluser]Phil Sturgeon[/eluser]
Right, this has been gone over many times in various forums but none of the answer seem to cut it.

Im currently working on two systems, one similar to Newgrounds (upload flash games) and another similar to YouTube (upload various types of movie files ready to be converted) so I need to somehow restrict to types easily. I currently have no bloody idea how to upload one single (potentially large) file without getting a timeout either, and this has to be done on a server with no installation access and PHP 4.4.x

Can I use the file upload class or ftp class?

I dont care if I use Java applets, AJAX/Flash or what, but I need to work out something shiny within the next few days or im toast! Anyone?

Edit: http://encodable.com/uploaddemo/ or http://store.masrizal.com/product/custom...mple_2.phplooks awesome, anyone know of something like this for PHP that is free?

[eluser]esra[/eluser]
Maybe using FIFO buffers. There should be some PHP4 and PHP5 classes on phpclasses.org for handling FIFO.

Take a look at how this is handled for the Docman component for Joomla 1.0.x which handles extremely large file uploads such as large pdfs and videos. I believe the general idea is to set the incoming and outgoing buffer sizes to a fixed value based on connection speed. That is, the buffers need to be large enough to keep pace with the current connection. Then its possible to upload specific amounts of data (fill the buffer with a portion of the incoming or outgoing file) before writing to disk, download or upload more until the buffer fills again, write to disk again (e.g., cocatenating to a temporary file for an upload), and so on. You might be able to find some classes that handle the requirement on phpclasses.org. Note that this could get fairly memory intensive on a popular site.

In the old days when bullentin boards were popular, this is how massive amounts of email (usually one gigantic file) was processed for networks like Fidonet/RBBSnet or gates such as ufgate which imported news to Fidonet/RBBSnet from USENET.

[eluser]Phil Sturgeon[/eluser]
Perhaps im in stupid mode but i'm not understanding that FIFO malarky. I dont need a que of anything I just want one user to upload one file and not be able to submit the form untill its done. That second link is perfect I just cant afford to spend the entire project budget on the commercial licence...

I also dont really understand how to restrict file types using the Upload class. I saw some nasty function called _getext which i dont think is particularly secure as all it does is explode and take the first bit after the first ".", which surely can be faked with "innocentfile.jpg.exe"

[eluser]esra[/eluser]
[quote author="thepyromaniac" date="1185255825"] Perhaps im in stupid mode but i'm not understanding that FIFO malarky. I dont need a que of anything I just want one user to upload one file and not be able to submit the form untill its done. That second link is perfect I just cant afford to spend the entire project budget on the commercial licence...[/quote]

Well, I'm a bit of a relic from the past (programming since the early 80's) and involved with computers since the early 70's.

What is the maximum file upload size you want to support? Will you ever need a form to handle multiple uploads (important)?

I will check through some DVD's here to find an older copy of Docman that used the original non-commercial license. I was not aware that the project went commercial especially with the recent news from Jinx (Joomla head honcho) about enforcing GPL (forcing GPL, but allowing commercialism if the sources remained GPL; general idea was for commercial developers to offer a support apparatus or other incentives such as svn access to the latest/greatest code with commercial licences.). Last heard, the community was discussing a commercial business model that fit in with GPL and was legal.

[quote author="thepyromaniac" date="1185255825"] I also dont really understand how to restrict file types using the Upload class. I saw some nasty function called _getext which i dont think is particularly secure as all it does is explode and take the first bit after the first ".", which surely can be faked with "innocentfile.jpg.exe" [/quote]

There is a config parameter in the upload.php config file for setting allowed file types:

Code:

$config['allowed_types'] = "gif|jpg|png";

My upload.php is currently this:

Code:

<?php  if (!defined('BASEPATH')) exit('No direct script access allowed');

$config['allowed_types'] = "gif|jpg|png";
$config['encrypt_name'] = FALSE;
$config['max_height'] = "768";
$config['max_size'] = "100";
$config['max_width'] = "1024";
$config['overwrite'] = FALSE;
$config['remove_spaces'] = FALSE;
$config['upload_path'] = "./uploads/";

?>

You could add your own parameters based on your own needs (you know that).

There are several optional config files mentioned in the user_guide, including upload.php, that are not included in the distribution, mainly because the usage of the config parameters could in many cases just as well be defined in a operation-specific controller (they were more controller-specific config values for uploads, email, ftp, image_lib, etc.). Unless you were using multiple modules that required those config files, it might not make sense to have them because of CI's lazy initialization approaches.

I'm logged in from my laptop from someone's house right now, but will take a look at the CI code when I get home. Mimes.php defines all mime types supported by CI and it can be modified to add more mime types. I will have to look at the code, but I believethat mimes.php establishes allowed file types and upload.php establishes a subset of those mime types (Derek can probably confirm or deny). You may have to add more mime types to support all of the file formats you need to support. Theoretically, you should have two levels of control over file types when you consider the existance of mimes.php and upload.php.

There were a couple of nice PN apps that supported multiple video and audio formats as well as playlists. I don't recall if they handled uploads, but they may have. I may have copies of these on backup CDs and will take a look. I have not used PN for about 3-4 years, but it seems likely that they still exist. At least one, runs under Mambo/Joomla. I will try to track them down.

At work, there are plans for writing upload and download modules but their priorities are a bit low at this point. I would not mind playing with this over the upcoming weekend if I don't have to work.

[eluser]n00bPhpCoder[/eluser]
How come youre so smart and im not I need to be like you hehe. Much to learn so little time

[eluser]Phil Sturgeon[/eluser]
Thanks for the help so far dude. I remember now that file uploading has simple extention restriction, I was getting confused witht he FTP class at the time sorry >.<

I only want to upload one file, will never need multiple. Just want a form with all the data (category, whatever, title, etc) and a single upload box, and using JS I need a way to not let the form submit untill a valid file is uploaded. Sounds simple eh?

[eluser]Tookings[/eluser]
As for the big files, I worked on a project for the now defunct sendover.com.

The system would allow uploads up to 2GB. Everything was PHP, except for the upload script itself which was Perl. It read a piece of input (say 10K) and then wrote each chunk to disk. With careful attention to the chunk size and guarding against memory leaks we could get a about a hundred uploads pending per storage server at once.

We never did get it to work in plain PHP though -- that one script was always Perl. We were working on an AJAX progress bar system before the site was shuttered.

From a quick search, it seems like there is more out there these days (http://encodable.com/filechucker/ seems to do > 2GB, and uses a little Perl piece as well to stream the upload. Seems to be a theme...). I've never that product before now, but looks fairly cheap.

Good luck! And let me know if you pull it off using only PHP.

[eluser]Phil Sturgeon[/eluser]
Thats the one I posted! :p I've talked one of the members here into helping out with this bit, hopefully using filechucker.

[eluser]Tookings[/eluser]
Heh. Wow. Needed more coffee this morning...sorry!

We used a script based somewhat on a free perl file online, and added lots of postback code to the bottom so that it would start a PHP script when it was done with an upload to do all the other work.

I couldn't find it in a search though... (I can probably find it if you're really interested in hacking up a messy Perl file. PM me if I should look -- but there is probably MUCH better out there somewhere.)

Sorry about the dupe postback of that link. :cheese: Filechucker does look promising though if you end up using it -- let us know how it goes.

[eluser]coolamit[/eluser]
well, over the years, I've also seen only Perl scripts that offer real-time progress bars, others are just sort of emulation. and yes, the size problem has also been there in PHP!! I'll also be interested in a solution if anyone finds about it!