First create three pages and place them in this path
C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\14\TEMPLATE\LAYOUTS\1033\
Custom_Error.html
Custom_NotFound.html
Custom_AccessDenied.html

Note: if you have multilingual site you need to place these pages in their language folder for example forArabic
C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\14\TEMPLATE\LAYOUTS\1025\

Second Open SharePoint 2010 Management Shell with Farm Administrator User
Now For Custom page Not Found runs the following Power Shell:
$webApp = Get-SPWebApplication http://test 
$webApp.FileNotFoundPage = “Custom_NotFound.html” 
$webApp.Update() 

For Custom Error Page, Access Denied page,.. All pages listed in the below image

Run the following Power Shell:

$webApp = Get-SPWebApplication http://test
$webApp.UpdateMappedPage([Microsoft.SharePoint.Administration.SPWebApplication+SPCustomPage]::Error,"/_layouts/1033/Custom_Error.html") 
$webApp.Update()