Tensors! *spawn*

I am sure I read some documentation/paper on this and if I get the time will try and find it, pretty sure some time ago I mentioned on here but going to be a pain to find.

 

Do you mean that each sub-core is divided in 4 groups of 4 FMAs basically? Would be very interested if there was anything hinting at that!

If you mean the Hotchips presentation on Volta, that split each SM in 4 sub-cores, but doesn’t split the sub-cores further IIRC

 

Hotchips comes to mind but there was also another document/paper on the subject, started to try and trawl through either what I was given or some of the papers/presentation I read, sadly like a needle in a haystack.
It was more on the operation-instruction you describe written/read twice data, I would not like to say if it explained too much detail on the implementation until I can find it - is bugging me

 

NVIDIA didn't announce any new architectures at GTC 2018, and I'm still interested in what analysis and speculation you can come up with.

 

Benchmark CIFAR10 on TensorFlow with ROCm on AMD GPUs vs CUDA9 and cuDNN7 on NVIDIA GPUs

Amazing results for Vega!

 

It does not seem to be using the Tensor cores if one looks at the results between V100 and Pascal GPUs; can be understandable as not everything can be reduced-optimised to fp16 but worth noting Google TPU2 is also only mixed precision so the direction and momentum is going that way.
Looking at other TensorFlow results the gap is much larger.
Table results at bottom for some general testing without optimisation: https://www.pugetsystems.com/labs/h...s-and-Testing-of-FP16-for-Deep-Learning-1141/

But yeah there are caveats to getting the most out of the Tensor cores and how used, and nice to see AMD improving from TF1.1, crux is how quickly they can get up to TF 1.6 support.

 

I think you're referring to something a research team at Citadel did, and they tracked down the tensor core threads/matrices/registers mapping (for their part, NV refers to the fragments' identity and locality as undefined). They did a presentation of this at one of the recent GTCs, though not Hot Chips IIRC.

I tried extrapolating from that paper and NVIDIA's various dev documentation on the topic, though I'm not confident in the resulting accuracy.

 

Thanks for the link Nate.
I recognise the names and seen some of their work before, maybe one of their earlier pre-publish papers without geometry aspects *shrug*.

 

Accelerating Reduction and Scan Using Tensor Core Units
November 23, 2019

https://arxiv.org/pdf/1811.09736.pdf