First you need to downgrade your ps3 by using what is called an E3 Flasher it is quite hard, so I suggest just pay someone who is an expert on downgrading a Ps3 console if you find videos on youtube saying you can downgrade a ps3 without the E3 Flasher, that is absurd it will brick your console

What the E3 does is downgrade you to Firmware 3.55, when on 3.55 you can install a so called CFW (Custom Firmware) aka a jailbreak. Once you have a jailbreak you can install homebrew. The modder/hacker on GTA you saw probably used an FTP program to connect his pc to his ps3, from there its easy to replace certain files in your ps3's HDD in his case he downloaded modded files and replaced them. I hope this helps you a bit

Inorder to use a modded eboot you have to have a CFW ps3 that can go online.
He was using a modded eboot or memory editing(RTM Tools)

The only way to use a modified EBOOT is through using a Custom Firmware PS3 console. Otherwise, you can use an E3 flasher to downgrade your system if it's a NOR console type. Or just search ebay or NextGenUpdate marketplace for a console sale.