I am working on a project regarding router's firmware. I analyzed several of the open source projects (OpenWrt,DDWrt,tomato,...) and used OpenWrt tools to build custom firmware for the router, but my requirement is to build it from scratch. For this I will first get to know how to use boot loader to load Linux kernel in my device and compress it as a .bin file so that it can be loaded into the router for testing purposes. Thanks in advance.
I don't think router is suitable for learning/development, because there is no JTAG, you can't flash firmware easily and router SoC are not for DIY (no documentation and support). Maybe it's better to try some Atmel/NXP/TI Linux SOC first and than apply this experience to your project.
But if you are OK to build system from scratch with your board read this: http://free-electrons.com/doc/training/embedded-linux/
It's very nice training about embedded linux. It will describe how to build u-boot, kernel and minimal RootFS.
You will not be able to run anything without RootFS and some libraries. Also it's not OpenWRT anymore if you want to make it from scratch without unnecessary stuff, so you no longer need OpenWRT tools, you can use Yocto or Buildroot (much simpler to learn).
Technically, if you have root access to the device, you simply need to figure out which
/dev/mtdX device holds the kernel and then you can flash it if you can get programs from
mtdtools running on the device. I think on most units one
/dev/mtdX device is for the kernel and another is for the rootfs. You need to find out the format both are in and package accordingly, which is something you can do from the router itself.
But, unless you know your kernel and rootfs work, it's bad to rely on a working OS on the router to do this, unless you 100% sure know your kernel and rootfs will work when the system is restarted.
So obviously you really need access to the router's bootloader, which depends on the make, model, and hardware revision of your router. The OpenWRT/DD-WRT sites can help you if you look through their installation instructions.
A very few routers (AR7 based ones) have an FTP server running for a few seconds before Linux is loaded, and you can load a new kernel and rootfs into the flash via FTP commands, even if your kernel/rootfs doesn't work.
However, the bootloader on many routers is only available via a hidden onboard 3v UART, which likely requires you to solder header pins somewhere on the unit. I think typically once you gain access to this it's possible to flash via TFTP.
However, you might look into emulating a small flash-based system with QEMU. I've never messed around with ARM or MIPS emulation much on QEMU but QEMU's ARM emulation includes emulation of hardware such as onboard flash, etc. This may be much easier than developing on a physical device.