Author Topic: why do we need mirrors?  (Read 2918 times)

0 Members and 1 Guest are viewing this topic.

Offline fanis

  • Jr. Mitglied
  • **
  • Posts: 65
  • Skill: Novice
why do we need mirrors?
« on: 12. September 2013, 06:37:37 »
Seriously!!!

Why do we need mirrors? This question puzzles me for more than a year I can not answer it myself and now it is the first time I speak out about it searching for a sane answer.

Instead of trying to maintain mirrors all over around the globe the "obvious" way that I see to distribute the packages would be a torrent-like protocol. I would propose as a client rtorrent that I have seen running at the background really effective. All this can be automatic. Magnet links have very low footprint and include SHA1 verification. I would say SHA1 is perfectly safe but of course if we want to be paranoid (I would not be about that) nothing forbids us to make a SHA2 verification also.

Now I know no distribution that does this. So maybe it is a really stupid idea, but I would like to know why.

Offline philm

  • Core Team
  • *****
  • Posts: 10671
    • Manjaro Linux
  • Branch: Unstable
  • Desktop: XFCE
  • GPU Card: Geforce GTX 650
  • GPU driver: non-free
  • Kernel: Maintainer - so all ;)
  • Skill: Guru
Re: why do we need mirrors?
« Reply #1 on: 12. September 2013, 07:37:49 »
You simply need mirrors all over the world cos it is the way to go. You can't use torrents for that. Torrents need seeders and everybody should be online. If we update some packages you have to download them to your local "torrent mirror". Also I don't think that you want to share 50 GB from your PC with all the world. Simple example: You are based in a town on the country side. What would you say if everybody gets the stuff from you, even if you have a really slow internet connection? And I don't think you have all packages downloaded on your end Manjaro offers. Please read this and the wiki.

Offline fanis

  • Jr. Mitglied
  • **
  • Posts: 65
  • Skill: Novice
Re: why do we need mirrors?
« Reply #2 on: 12. September 2013, 09:39:17 »
thank you philm for an answer but I am still not satisfied.

Quote
Torrents need seeders and everybody should be online.
of course good seeders will be needed like now we need mirrors. So I see no difference here except that if someone wants to seed he can simple seed, no need to setup a mirror. When there is a traffic about some new packages everyone who downloads will help the system and I keep believing that torrents are a way more reliable way in many ways.

Quote
Also I don't think that you want to share 50 GB from your PC with all the world.
there going to be needed some good seeders (as I said above) to do that. But most people can even delete the images as soon as the packages are installed if they want. (This can be done automatically)

Quote
Simple example: You are based in a town on the country side. What would you say if everybody gets the stuff from you, even if you have a really slow internet connection?
From my experience I do not see that this example makes any sense. When you download from torrent you do not download from one person. You download from everybody that offers bits of what you ask. Anybody who seeds only helps and does not slow down the system in any way. If he is fast seeder, he helps a lot, if he is a slow seeder he helps only little but he only helps.

Offline ringo

  • Held Mitglied
  • *****
  • Posts: 9684
  • www.manjaro-nl.org
    • Manjaro Belgium + forum
  • Branch: unsourced
  • Desktop: xfce
  • GPU Card: Nvidia Geforce 8600 GT (acer 23inch)
  • GPU driver: Free*
  • Kernel: linux kernel
  • Skill: Novice
Re: why do we need mirrors?
« Reply #3 on: 12. September 2013, 10:19:26 »
dont see te point to 'seed' the Mirrors.

there is an quality point and some stability..

dont think so needed personally.. its a system offcourse but seeding would change a lot in packagemanagement and is buggy to...

i would not seeed that..

dont think linux distro general does use seedings? linux general is watered down on some point.. but seedings is not a good thing.. when you have to seed you have first 50gb to seed, straight syncing is alread much hours and with torrent doesnt mean it would be faster...

Mirrors are general safe offcourse, but if you have an Rotten apple between te seeds on somuch packages that is not good :)
Dutch go the dutch site :)

Offline llogg

  • Sr. Mitglied
  • ****
  • Posts: 421
  • Skill: Novice
Re: why do we need mirrors?
« Reply #4 on: 15. November 2013, 19:31:32 »
thank you philm for an answer but I am still not satisfied.
of course good seeders will be needed like now we need mirrors. So I see no difference here except that if someone wants to seed he can simple seed, no need to setup a mirror. When there is a traffic about some new packages everyone who downloads will help the system and I keep believing that torrents are a way more reliable way in many ways.
No, having seeders out there with individual control over the packages they are offering is not a more reliable system than having a centralized repository from which everyone can get the desired packages with reasonable assurance about security, quality, and compatibility.
Quote
there going to be needed some good seeders (as I said above) to do that. But most people can even delete the images as soon as the packages are installed if they want. (This can be done automatically)
This is not really an argument in favor of torrents.
Quote
From my experience I do not see that this example makes any sense. When you download from torrent you do not download from one person. You download from everybody that offers bits of what you ask. Anybody who seeds only helps and does not slow down the system in any way. If he is fast seeder, he helps a lot, if he is a slow seeder he helps only little but he only helps.
The point is about his experience.  If he is seeding to a lot of people his bandwidth will get quickly eaten up.
Laptop: Ideapad U310, Kernel 4.1, 4.3, MDM, XFCE

Desktop: Kernel 4.0, 4.1, MDM, XFCE, Nvidia non-free drivers

Offline Neil

  • Neuling
  • *
  • Posts: 17
  • Branch: Stable
  • Desktop: XFCE
  • GPU Card: Intel Extreme Graphics 2
  • GPU driver: Kernel driver - i915
  • Kernel: 3.10
  • Skill: Novice
Re: why do we need mirrors?
« Reply #5 on: 15. November 2013, 20:02:01 »
fanis, are you talking about ISO downloads, or repository mirrors?

I know of several distros that use torrents to download the ISO, but I don't see how it could work with the repositories.

Offline ringo

  • Held Mitglied
  • *****
  • Posts: 9684
  • www.manjaro-nl.org
    • Manjaro Belgium + forum
  • Branch: unsourced
  • Desktop: xfce
  • GPU Card: Nvidia Geforce 8600 GT (acer 23inch)
  • GPU driver: Free*
  • Kernel: linux kernel
  • Skill: Novice
Re: why do we need mirrors?
« Reply #6 on: 15. November 2013, 20:04:54 »
if someone has an vps server it can start an torrent :)

torrent is also on people so they should look in the manjaro discussion and take a look in the torrent topic. if new version is there take it by torrent and seed it for the one that want.


updating by torrent is buggy one bad machine and have all bugs

is like using powerpils that using all the mirror by updating !
one bad mirror and you have been doomed !
:)
Dutch go the dutch site :)

Offline Ayceman

  • Core Team
  • *****
  • Posts: 2113
  • Security Chief :P
  • Branch: stable, unstable
  • Desktop: Xfce
  • GPU Card: GTX780/HD3000/AMD8400
  • GPU driver: Nvidia/Intel/Catalyst
  • Kernel: latest stable+lts (64)
  • Skill: Advanced
Re: why do we need mirrors?
« Reply #7 on: 17. November 2013, 14:46:48 »
And I don't think you have all packages downloaded on your end Manjaro offers.

THIS.

A torrent system would actually require full-time seeders that have all the packages, which amounts to more infrastructure than mirrors.

Offline Zediker

  • Neuling
  • *
  • Posts: 13
  • Branch: stable
  • Desktop: Xfce
  • GPU Card: nvidia 660Ti / nvidia Optimus 540M
  • GPU driver: 325.x / Bumblebee
  • Kernel: 3.10.x
  • Skill: Intermediate
Re: why do we need mirrors?
« Reply #8 on: 02. December 2013, 23:18:12 »
actually, you could create a Distributed Hash Table for the repos, or should i say "the repo". You could still have the mirrors as well as "master" nodes with whichever node Phil wants as "supreme node(s)". It would operate like a combination of git, bit-torrent, and kademlia (though, i guess thats what really powers bit-torrent). Updates would be to some "master manifest" that is easily locate-able on the overlay network that specifies hash-keys of the updates to be located, retrieved, and applied. This manifest and subsequent files would also be signed by a master key to ensure non-repudiation and integrity, etc. Additionally, no one node would "need" all the repo files, just whichever files' hash-keys (or file segment hash-keys) are closest to their node in the key-space. This according the the kademlia researchers provides "protection" against nodes entering and leaving.

Its all doable and would work, you just cant guarantee time-of-delivery to the network and during the early stages of pushing updates to the overlay network, there wouldnt be much difference than the current mirror system, but after the network has been sufficiently seeded, it could be faster and would allow for segmented file storing/retrieval. It would also allow for an interesting way for "approving" non-core packages or creating an AUR-like mechanism...
« Last Edit: 02. December 2013, 23:24:59 by Zediker »

Offline thundersqueak

  • Held Mitglied
  • *****
  • Posts: 1755
  • everyone dies
  • Branch: all of em
  • Desktop: gnome
  • GPU Card: nvidia 9300
  • GPU driver: non-free
  • Kernel: most,if not all
  • Skill: Novice
Re: why do we need mirrors?
« Reply #9 on: 02. December 2013, 23:50:20 »
yep that's what i think,the manjaro devs don't have enough to do let's get them to change the infrastructure as well  :P .On a serious note this is the kind of discussion that needs to wait until 1.0 is a bit closer don't you think?
"Why is it that every grubby little government i pass has the impertinence to assume that i can spare the time and energy necessary for their destruction?"

Offline Rob

  • Core Team
  • *****
  • Posts: 5390
  • Alias: Korrode
  • Branch: Unstable
  • Desktop: Xfce+Compiz
  • GPU Card: nVidia GTX560Ti
  • GPU driver: non-free
  • Kernel: linux314 x64
  • Skill: Advanced
Re: why do we need mirrors?
« Reply #10 on: 02. December 2013, 23:57:27 »
Damn p2p fanbois.

HTTP mirrors are fine. Good in fact. Very versatile.

Offline QtAndNice

  • Jr. Mitglied
  • **
  • Posts: 93
  • Branch: stable
  • Desktop: KDE 5.x.(current stable)
  • GPU Card: nVidia GTX 760
  • GPU driver: non-free
  • Kernel: 4.1.18-1-lts-tomoyo (AUR)
  • Skill: Novice
Re: why do we need mirrors?
« Reply #11 on: 03. December 2013, 00:00:14 »
It is highly unrealistic to use such system on a broad scale, the amount of IO a seeder would get and the RAM used to pre- cache  the most accessed files would easily use up all the resources on the seeder's computer slowering he's hard drive to a stand still, not to mention the CPU load for managing all the transactions/ caching and hashing from all over the world.

Multiple casual Manjaro users: oh let's make an pacman -Syu
Seeder(playing tux racer): ... hmm ?, i didn't pressed pause... OMG DDOS attack !
« Last Edit: 03. December 2013, 00:07:00 by QtAndNice »

Offline Zediker

  • Neuling
  • *
  • Posts: 13
  • Branch: stable
  • Desktop: Xfce
  • GPU Card: nvidia 660Ti / nvidia Optimus 540M
  • GPU driver: 325.x / Bumblebee
  • Kernel: 3.10.x
  • Skill: Intermediate
Re: why do we need mirrors?
« Reply #12 on: 03. December 2013, 01:03:39 »
I think the mirriors are fine, I'm just saying that the statements of it cant be done are inaccurate. Additionally, you would want to put some sort of throttling mechanism in place for exactly what you were saying, and something like that is trivial in comparison to the code related to managing the overlay node. Additionally, you probably wouldn't pre-cache, but instead segment the files to a designated size (say 65KB) or some common multiple of the standard TCP/IP window size and just run ad-hoc on requests, denying when a load threshold is exceeded. An additional benefit of the segmentation is that one single node wouldnt be hit for the full "size" of the file being re-assembled, just the size of the segment. It would in a way, enforce balance over the overlay network as the hash sums from sha1, 256, or 512 should be "uniformly random".

Offline Lukimya

  • Held Mitglied
  • *****
  • Posts: 884
  • Branch: Unstable
  • Desktop: Gnome 3.20x
  • GPU Card: Intel HD4400
  • GPU driver: free
  • Kernel: 4.4.5-1-ARCH
  • Skill: Intermediate
Re: why do we need mirrors?
« Reply #13 on: 03. December 2013, 03:41:42 »
I think my ip would be banned with that much p2p traffic. Torrent has advantages but it is better suited for example backing up your own sytem than for replacing the mirrors :)

Offline Verändert2.0

  • Held Mitglied
  • *****
  • Posts: 1890
  • Ni Dieu ni maître
  • Skill: Novice
Re: why do we need mirrors?
« Reply #14 on: 03. December 2013, 07:14:23 »
Never touch a working system. And http mirrors work