i have two torrents that are massive (each around 40GB)

i inadvertently (don't ask) started downloading them on two separate machines

they're both at around 45% completion

since BT, by nature, downloads pieces randomly, in theory, each file should have at least some pieces that the other copy does not have

is there any way i can merge these together? i tried diff, but it only does text files.

9/23/2008 10:59:47 PM

bindiff

9/24/2008 12:23:43 AM

You should be able to write a program somewhat easily to do this...

9/24/2008 1:04:13 AM

yeah, but i'd rather not reinvent the wheel if one already exists

bindiff looks promising, thx2u nic

9/24/2008 1:36:48 AM

40 GB

good god what are you downloading

9/24/2008 7:51:50 AM

don't ask, don't tell

9/24/2008 9:40:05 AM

damn you evan, stealing my bandwidth

9/24/2008 9:41:36 AM

it's one of those SUPER MEGA ALL MY FONTS 4 U IN THE WORLD.torrents

9/24/2008 9:41:57 AM

rainbow tables?

9/24/2008 9:43:00 AM

he is dling pron

9/24/2008 9:49:16 AM

no pr0n

no rainbow tables

no fonts

9/24/2008 10:06:05 AM

I downloaded a DS rom pack that was over 50 gig, something like that?

9/24/2008 10:08:40 AM

dl all the will and grace episodes?

9/24/2008 10:12:12 AM

im gonna guess that he short answer to this is no

long answer, noooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo

Quote :

"since BT, by nature, downloads pieces randomly, in theory, each file should have at least some pieces that the other copy does not have"

not always. if there's only 45% of that file available out there (common), they'd have the same pieces

9/24/2008 10:13:53 AM

Quote :

"smoothcrim rainbow tables?"

Is WPA pretty easy to crack now? I downloaded some tables awhile back, but gave up on my cracking endeavor because I needed to fab up an antenna hookup for my pcmcia card to pull in the signals better.

9/24/2008 10:14:46 AM

emulation?

9/24/2008 10:41:43 AM

Quote :

"not always. if there's only 45% of that file available out there (common), they'd have the same pieces"

there were 10.2 copies of the file available in the swarm amongst 104 seeds

9/24/2008 10:56:28 AM

well can you do anything with each of the 45% incomplete files you have so far...can you actually use any of the data?

if the answer is no then how do you expect to do anything with a 60%/70%/80% incomplete torrent file if you were somehow able to miraculously piece them together.

9/24/2008 11:04:28 AM

Quote :

"don't ask, don't tell"

Gay porn.

EDIT:

Ok, on a serious note. Is it one large file, or several files? You should be able to see the percentage completion for each file. Seems like it would be simple (though arduous) to do a replace on set 1 of all the files in set 2 that are more complete. This obviously wouldn't net you a new set that had all the info from both original sets, but it might give you a leg up.

[Edited on September 24, 2008 at 11:14 AM. Reason : -]

9/24/2008 11:11:05 AM

Quote :

"if the answer is no then how do you expect to do anything with a 60%/70%/80% incomplete torrent file if you were somehow able to miraculously piece them together."

when i finish downloading the files, i won't have to download as much.

9/24/2008 11:16:26 AM

Simpsons seasons 1-19

[Edited on September 24, 2008 at 12:45 PM. Reason : -]

9/24/2008 12:44:33 PM

Could be a raw blu ray movie.

9/24/2008 1:38:27 PM

synapse is pretty lost in this thread, huh?

9/24/2008 3:46:57 PM

i think it's funny how you guys are more interested in what it is i'm downloading

9/24/2008 5:28:57 PM

Quote :

"somehow able to miraculously piece them together."

i don't quite see how it would be miraculous

i don't think you quite know how bittorrent works... my torrent client preallocates the space by creating a file full of zeroes. it then asks the tracker for peers/seeds to connect to and which pieces they have available. it downloads pieces (mostly in random order, but it tends to prioritize the rarest pieces first) from various peers and compares the sha1 hash of the downloaded piece to the sha1 hash stored in the torrent. if they match, it inserts the data at the appropriate location in the file (it knows where it goes because all pieces are the same size defined by the torrent file).

therefore, statistically speaking, my two files will indeed have some of the same information, but a significant portion of it will be different. something like bindiff will work because with pieces that differ between the files, one file will have 512kB of zeroes and the other will contain the data. since the torrent client only inserts the data if it passes the hash check, pieces that exist in both files will be the exact same piece of 512kB data, and, therefore, would not be included in the diff.

apply the diff to one of the files, and you have a composite representation of the pieces available in each file. tell your torrent client the location of the new converged files and have it re-run the hash check.

understand it now?

[Edited on September 24, 2008 at 5:41 PM. Reason : .]

9/24/2008 5:40:43 PM

I think he doesn't understand that you're trying to merge two incomplete torrents before continuing the download. He seems to think that you're trying to merge together two overlapping 45% complete torrents and somehow end up with a 100% complete download.

9/24/2008 5:48:15 PM

oh, i see

nobody in their right mind would assume that, however
45 + 45 doesn't even equal 100.

9/24/2008 5:55:06 PM

Yeah, well...

Quote :

"if the answer is no then how do you expect to do anything with a 60%/70%/80% incomplete torrent file if you were somehow able to miraculously piece them together."

durr

9/24/2008 6:02:58 PM

alldata

9/25/2008 1:43:40 AM

I'm not well versed in torrents, but couldn't you just get your 2 machines together and get them to sync with what they have via your torrent client? Maybe disable downloading on the one you don't want the complete torrent on?

9/25/2008 12:32:41 PM

^ That actually could work. Azureus has an option to priortize peers on your LAN, and you can add a peer by IP address. That way, you're kind of "slipstreaming" one torrent in without worry about the whole bindiff process.

9/25/2008 12:42:15 PM

When bittorrent clients allocate the entire space for a torrent, does it 0 out the whole file first?

9/25/2008 1:41:35 PM

Quote :

"He seems to think that you're trying to merge together two overlapping 45% complete torrents and somehow end up with a 100% complete download."

uh yeah you got me there

i totally see how you could get that from this line

Quote :

"how do you expect to do anything with a 60%/70%/80% incomplete torrent file if you were somehow able to miraculously piece them together."

Quote :

"when i finish downloading the files, i won't have to download as much."

that makes sense

Quote :

"i don't quite see how it would be miraculous"

well let me know when its done then, and how you did it.

9/25/2008 2:20:03 PM

this is why newsgroups uses lots of .rars.

9/25/2008 2:23:40 PM

Quote :

"When bittorrent clients allocate the entire space for a torrent, does it 0 out the whole file first?"

depends on the client, in utorrent, you can specify whether you want it to preallocate or not

Quote :

"alldata"

i already have it
i guess i should post in the garage thread then

[Edited on September 25, 2008 at 2:26 PM. Reason : .]

9/25/2008 2:25:07 PM

^^ original implementations of bittorrent required breaking files up in to rars too.

9/27/2008 11:07:25 PM

^^yoo, i need some help with alldata. i have the program and the dvd i want..problem is it says the program is the wrong version for that dvd

so, what version stuff do you have? i either need to upgrade the program or downgrade the data dvd's (need the ford one)

9/28/2008 12:02:05 AM

Quote :

"I'm not well versed in torrents, but couldn't you just get your 2 machines together and get them to sync with what they have via your torrent client? Maybe disable downloading on the one you don't want the complete torrent on?"

I was going to suggest this too. It seems like the simplest option, if its an option at all.

9/28/2008 10:36:11 AM