Indeed..
Interestingly, we were having this discussion (in a similar form) with respect to a radio telescope at work - modern arrays like SKA, LOFAR, MeerKAT, etc. have *lots of data* being pushed around. I'm working on one that flies in space (so we're not going to have a fiber to the ground<grin>) - but there has always been discussion of the difference between "real time low latency interconnect" and "long latency interconnect"
One aspect of this, though, is that buying and installing fiber is a "capital expenditure", while buying lots of Snowmobiles (or mailing lots of diskdrives) is a "operating expense" and often comes out of a different bucket of money. Of course, if you can procure your network services " by the month" or "by the bit" then someone else deals with the capital expense.
This comes up for us in NASA missions and projects all the time - do you buy test equipment or rent it? And this is where you can make the correct decision for the short run that may turn out to be more expensive in the (speculative) long run. Consider Spirit and Opportunity - minimum mission was about 3 months, goal was 1 Martian year (I think). And here we are 10 years later with Opportunity still grinding along (well, not right now, because of the dust storm blotting out the sun). Curiosity - 1 Martian year (about 2 Earth years), and it landed in 2012.
On 7/27/18, 12:11 AM, "Beowulf on behalf of Jörg Saßmannshausen" <beowulf-***@beowulf.org on behalf of sassy-***@sassy.formativ.net> wrote:
Hi all,
Jim: the flip side of the cable is: once it is installed you still can use it,
whereas with the snow mobile you have to pay for every use.
So in the long run the cable is cheaper, specially as we do need fast
connection for scientific purposes.
I was at a talk here in London not so long ago when they were talking about
data transfer of the very large telescope. As that is generating a huge amount
of data a week, say, a snow mobile would simply not be practical here.
Besides, the data is generated on literally the other side of the world.
All the best from a hot, sunny London
Jörg
Post by Lux, Jim (337K)A quick calculation shows that the bandwidth is on the order of single digit
Tbps, depending on the link length and road conditions. Pasadena to Ann
Arbor works out to 7.6 Tbps on I-80
If they charge by fractional months - it's about a 33 hour drive, so call
that 1/15th of a month. So about $35k to do the transport. Significantly
cheaper than 4000 km of fiber, coax, or cat 5 cable.
Jim Lux
(818)354-2075 (office)
(818)395-2714 (cell)
-----Original Message-----
Youhanaie Sent: Thursday, July 26, 2018 5:10 PM
Subject: Re: [Beowulf] Lustre Upgrades
Yep, this could be considered as a form of COTS high volume data transfer ;-)
from https://aws.amazon.com/snowmobile/faqs/ (the very last item)
"Q: How much does a Snowmobile job cost?
"Snowmobile provides a practical solution to exabyte-scale data migration
and is significantly faster and cheaper than any network-based solutions,
which can take decades and millions of dollars of investment in networking
and logistics. Snowmobile jobs cost $0.005/GB/month based on the amount of
provisioned Snowmobile storage capacity and the end to end duration of the
job, which starts when a Snowmobile departs an AWS data center for delivery
to the time when data ingestion into AWS is complete. Please see AWS
Snowmobile pricing or contact AWS Sales for an evaluation."
So it seems a fully loaded snowmobile, 100PB at 0.005/GB/month, would cost
$524,288.00/month!
Cheers,
Fred.
Post by Lux, Jim (337K)SO this is the modern equivalent of "nothing beats the bandwidth of a
station wagon full of mag tapes" It *is* a clever idea - I'm sure all the
big cloud providers have figured out how to do a "data center in shipping
container", and that's basically what this is.
I wonder what it costs (yeah, I know I can "Contact Sales to order a
AWS Snowmobile"... but...)
Jim Lux
(818)354-2075 (office)
(818)395-2714 (cell)
-----Original Message-----
Sent: Tuesday, July 24, 2018 11:21 AM
Subject: Re: [Beowulf] Lustre Upgrades
https://aws.amazon.com/snowmobile/
They drive a 45-foot truck to your data centre, fill it up with your
data bits, then drive it back to their data centre :-()
Cheers,
Fred
Post by Jonathan EngwallSnowball is the very large scale AWS data service.
Post by John Hearns via BeowulfJoe, sorry to split the thread here. I like BeeGFS and have set it up.
I have worked for two companies now who have sites around the world,
those sites being independent research units. But HPC facilities are
in headquarters.
The sites want to be able to drop files onto local storage yet have
it magically appear on HPC storage, and same with the results going
back the other way.
One company did this well with GPFS and AFM volumes.
For the current company, I looked at gluster and Gluster
geo-replication is one way only.
What do you know of the BeeGFS mirroring? Will it work over long
distances? (Note to me - find out yourself you lazy besom)
This isn't the use case for most/all cluster file systems. This is
where distributed object systems and buckets rule.
Take your file, dump it into an S3 like bucket on one end, pull it
out of the S3 like bucket on the other. If you don't want to use
get/put operations, then use s3fs/s3ql. You can back this up with
replicating EC minio stores (will take a few minutes to set up ...
compare that to others).
The down side to this is that minio has limits of about 16TiB last I
checked. If you need more, replace minio with another system
(igneous, ceph, etc.). Ping me offline if you want to talk more.
[...]
_______________________________________________
Computing To change your subscription (digest mode or unsubscribe)
visit http://www.beowulf.org/mailman/listinfo/beowulf
_______________________________________________
Computing To change your subscription (digest mode or unsubscribe)
visit http://www.beowulf.org/mailman/listinfo/beowulf
_______________________________________________
change your subscription (digest mode or unsubscribe) visit
http://www.beowulf.org/mailman/listinfo/beowulf
_______________________________________________
To change your subscription (digest mode or unsubscribe) visit
http://www.beowulf.org/mailman/listinfo/beowulf
_______________________________________________
Beowulf mailing list, ***@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
_______________________________________________
Beowulf mailing list, ***@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit http://www.b