Yeah that's exactly how I feel about it too. By definition a delay means it isn't good enough to bring to market in its current state. So it has to catch up to "good enough" status first. Now you could say that it's due to yields/leakage/etc but those things are usually indicative of performance levels too - i.e gotta lower clocks to improve yields and lower power consumption.
NVIDIA Fermi: Architecture discussion
- Thread starter Rys
- Start date
Yeah that's exactly how I feel about it too. By definition a delay means it isn't good enough to bring to market in its current state. So it has to catch up to "good enough" status first. Now you could say that it's due to yields/leakage/etc but those things are usually indicative of performance levels too - i.e gotta lower clocks to improve yields and lower power consumption.
Groo The Wanderer
Regular
How many of them were Google/Amazon/AmazonEC2/Yahoo/WhateverMSbrandsthingsthishour type installations?
-Charlie
Groo The Wanderer
Regular
thats the exact world i live in and i haven't seen much different of late, i call BS on 1 socket installs in DC's it makes no sense. If you guys realised the cost per sq meter of a datacenter you will quickly see density is king. there are so many fronts it doesn't make sense on.
its inefficient in terms of power consumption from both increased inefficientices of AC to DC conversion in 1p servers vs blades and redundant parts within each server.
its very low density, i cant get 576 opertion cores in a 40 RU rack right now vs 14.4 racks to do that as 1 socket servers.
very costly in terms of localized cooling , for example i would need one APC 1/2 rack ac unit for 3 blade enclosures in a 40 RU rack, i would need 3-4 for 14.4 racks and much bigger water pumps.
given a 2 socket blade with 64 gig of ram with 2 istanbul cpu's is around 8-10K (AUD) single socket servers dont make sense. Sure there is hard disk to consider but most things in this space local disk doesn't have the performace anyway.
PS. im a network "engineer" in a DC
edit: lol, i forgot completely about networking costs, with 1 socket servers thats pretty much forcing you to go top of rack switching, with blades you can go something like a catalyst 6500 or nexus 7000 and just use fibre back to a couple of central points. assuming a redundant network design it is significatly cheaper as well as far more scalable to centralise your network access layer.
You seem to be assuming that your problem is compute bound, and from what I have seen, that is not the case for Google and many other similar installations.
-Charlie
Groo The Wanderer
Regular
I would disagree with this point slightly, you seem to be saying that there is nothing to blades but shared PSU/NIC/IO etc. I can think of one thing that a blade does better than a equivalently dense set of rack server, management.
If you have a full management scheme that is data-center wide and covers every bit of the systems, then there is little difference. If you don't, blades can be a bit easier to manage, and with the scale of some of these farms, a bit to a seriously high power adds up.
Then again, all the farms I know of don't manage as much as rip and replace anything that twitches wrong, it isn't worth the time or effort.
-Charlie
That is correct. It could also be a FB-DIMM.
And this is not OT because this clears up some misconceptions about FERMI use of memory when ECC is enabled.
aaronspink
Veteran
If this was 20 years ago, or possibly even 10 years ago, this could be a reasonable thing to consider. But given Nvidia's size and cost structures and given the HPC market, it is unreasonable for any reasonable executive to think that they can treat HPC as a primary and the broader GPU market as a secondary. The simple issue is that given the market functionality requirements in the consumer space, it would be hard if not impossible to create enough product differential between the HPC SKU and the game SKU to get a significant amount of the HPC crowd to pay 5-10x higher costs. There is little in the support realm that can reasonably support that cost differential.
And Ion for all intents and purposes is a dying product as well, which leaves them with Tegra. The fundamental problem with chipsets for any non-cpu vendor is that simply due to the natural progress of silicon, chipsets are a dying commodity. Though, I don't think we'll ever see notebook/desktop/server CPUs integrate the southbridge functionality simply because their is little to no engineering nor financial incentive to do so, in fact there is both engineering and financial incentives not to integrate.
The problem is that the mainstream is called mainstream for a reason. History is littered with failed HPC companies that had products very very similar to Nvidia's GPUs. In order to make the HPC market viable for Nvidia, they need to be competitive enough so that the mainstream market basically can pay all the R&D. If they can't, then their HPC product dies too.
aaronspink
Veteran
Unless they are using some pretty exotic DRAMs, their GDDR5 isn't calculating any ECC for the memory. Also Opterons do not use an "ECC" chip on the DIMM.
aaronspink
Veteran
ECC hasn't been done with 9b DRAMs since the dark ages (dark ages being defined as the area of computing modern DIMMs). Given a standard 64b width DIMM that uses either 16b, 8b, or 4b (though 4b is becoming an exceedingly rare and costly thing), it makes they most engineering and economical sense to add a 9th 8b dram to a 64b wide dimm to get 72b of data transmission.
Fermi most likely is doing 2 seperate memory accesses when it is using ECC. First access to the data and then a secondary access to a reserved ECC storage space. Given reasonable memory locality the costs would be roughly 1 additional access per every ~30 reads. And given an ECC write cache, 1 additional access every ~30 writes.
aaronspink
Veteran
Actually ECC provides quite a wide range of both correction and detection. Most high end servers for instance are 2b correct, 3b+ detect.
That doesn't explain the lower capacity when ECC is enabled.
The reserved ECC storage space would be allocated out of the memory pool..
aaronspink
Veteran
Except that in cases of bad data locality it has increased bandwidth costs. In addition, it would require DRAM chips capable of higher prefetch as well. ECC is done as it is done for current day systems primarily because of cost. The DRAM chips are exactly the same, the boards they go on is a minor cost factor, and the performance difference effectively doesn't exist. Doing interleaved ECC/Data would add either increased costs, reduced performance, or both.
For Fermi, I think it makes the most sense if they simply have a contiguous reserved section of the DRAM map for the ECC storage which enables a variety of optimizations no reasonably doable with an interleaved mapping.
You need extra space to store those ECC data, so capacity is lower (the second reserved area). As in ECC DIMM, 9 chips is used instead of 8 chips on normal DIMM. So, supposed that you want to do the normal 2b detect/1b correct ECC, you will have to reserve one byte for every 8 bytes, thus losing 1/9 capacity.
aaronspink
Veteran
I think I'm pretty right. It's called an EOS (ECC On SIMM) chip and. You'll find that any A brand (not Dell
Check this DIMM and notice the small IC in between the two rows of Memory chips. That's the ECC module.
Um..........
NO and NO!
no ecc calculations are done on the DIMM of any modern day computers. Even in the dark ages it was an utterly small minority of the ECC dimms.
aaronspink
Veteran
They were used for motherboards/systems that had no memory error detection (basically all non-server systems). For a vanishing short period of time, some insane people thought it was a reasonable retrofit to enable non-server systems to be used as servers. Luckily, those people are no longer employed, probably a consequence of spending 2x the cost on specialty ram for a system that was barely any cheaper to begin with.
aaronspink
Veteran
Um, yeah it does. If you are having to store an additional, say 8b per 64, 9b per 128, 10b per 256, or 11b per 512 of ECC data then you have to store it somewhere. Unless they have additional data lanes and some exotic way to get extra data lines not revealed, then they would have to store the ECC codes in the normal memory pool. This would require the hardware/drivers to map out sections of the DRAM for ECC code storage.
You don't see this on normal systems because they utilize 72b memory channels in which the additional 1/9th memory capacity is never actually advertised. For various reasons, Nvidia doesn't have this luxury.
Yep, but I was following on from your comment - "it makes they most engineering and economical sense to add a 9th 8b dram to a 64b wide dimm to get 72b of data transmission. Fermi most likely is doing 2 seperate memory accesses when it is using ECC. First access to the data and then a secondary access to a reserved ECC storage space."
Thought you were implying that this would be the setup in Fermi which would be contrary to what we know. I see now that the two statements were unrelated.
Thought you were implying that this would be the setup in Fermi which would be contrary to what we know. I see now that the two statements were unrelated.
Well none actually, because like I said before to that other guy, and you happen to reiterate in your very next post, all but ec2 (and that one is arguable) are not examples of HPC.
Do you think that Tesla's primary purpose is to compete for your webserver business, or your database?
I dont think they believe HPC will replace Graphics, more complement it similar to Professional providing profit while Graphics keeps the volumes high ie in economics an example of cross subsidization . The problem with graphics currently is the profitability appears to be progressively declining, rather than sit back they are trying to find new markets to compensate for this.
This chinese site a few months ago had some figures for the total HPC market i think from Nvidia's mid year presentation to analysts on Tegra and Telsa(about half way down under the S1070 picture):
http://news.mydrivers.com/1/145/145916_1.htm
So larger than $1bn in total. Now you just need to work out what percentage of the above nvidia figures it can get
Random accesses of size 1-48 bytes get their ECC for free with interleaved ECC bandwidth wise (burst 8, 64 bit bus). It's normal ECC which wastes bandwidth. There is only this very small region from 48-64 byte accesses (and their multiples) where it wastes any significant amount of bandwidth (and relatively little at that).
Why exactly? I see no problem using burst length 8 ... the memory controller simply caches bytes from an incomplete 8/1 pair for a bit to be able to handle sequential accesses, if they weren't needed in the end you can just drop them.
Oh man, now THAT would fuck you up when you have bad data locality.
