Isn't the DS a portable N64 with some tweaks?
No. DS's hardware is not based on N64's hardware. The N64 was a MIPS architecture, while DS is ARM. It's basic data throughput specs are similar (4MB of RAM, 120K polys/sec max), which is why games have roughly the same graphical fidelty as N64 games. Heck, I'd like to see the best game you could do on an Xbox 360 while limiting yourself to 4MB.
http://en.wikipedia.org/wiki/ARM_architecture
http://en.wikipedia.org/wiki/MIPS_architecture